Entropy is a measure of the disorder of a system.

The second law of thermodynamics states that in a closed system, the entropy of that system increases over time.

The name comes from the Greek meaning a "turning inward". The word was coined in 1865 by German physicist Rudolph Clausius as he needed a word to describe the "transformational content" of what he was observing/studying.

There are two fundamental physical laws of thermodynamics that govern the physics of the world in which we live.

• The 1st law of thermodynamics says that mass-energy can not be created or destroyed, but can change its form.



• The 2nd law of thermodynamics says that the entropy of a closed system increases over time.

Systems that, if they existed, would violate the 2nd law:

• perpetual motion machine (you cannot get a patent on such a design)

How does entropy relate to software development?

Physical entropy is the primary way in which the elusive concept of physical "time" is defined.

It turns out to be very difficult to precisely define digital "time" in a computer. That is a subject for another "time".

A software system over time can be thought of in terms of entropy.

Over time, the disorder of your software, as it fits into a system, increases. It is far more likely that any change to the system will make your software stop working than work better. Explain how entropy fits into the laws of physics and how the concept of entropy fits into software development.

Does your software ever randomly evolve into better software without a lot of intelligent effort on your part?

When your program does not work, does making random changes improve the chances of it working, or is it more likely that your program will less well after making the changes?

The statistics are against random changes to a program or to software improving the program or software.

Data (and information) can be represented using symbols that are made up of bits - zeros and ones in a binary code used to represent such data (and information).

A set of symbols (data) has a measure called entropy that approximates the lower bound on the number of bits needed to encode those symbols.

Such entropy constitutes a lack of order or a lack of predictability.

The following Shannon entropy formula expresses how entropy for a set of symbols is calculated.

• S is the entropy
• P_i is the probability of a given symbol i in a sequence of symbols appearing

Claude Shannon, in the 1950's, used the term "information" to describe the statistical properties of data transmitted from one place to another. He was working for the telephone company AT&T at Bell Labs in order to improve the quality of phone calls. He founded the modern field of (statistical) information - he was not overly concerned with the meaning of the "data" that he called "information". Previously, the term "information" was used as the verb "to inform". That is, one is "informed" that such and such is true - what we now call "information".

Whenever the mathematical genius John Von Neumann (conventional computer architectures are called Von Neumann machines) was asked by Claude Shannon what to call a measure of the lack of content in data, Von Neumann is reported to have told Shannon to call it "entropy" because then nobody would understand it.

Von Neumann to Shannon:

You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.

Note that the concept of information developed by Shannon has to do with statistical properties of data.

It does not have anything to do directly with the meaning of the data. That is, information.

So what is called "information theory" might be better called "data theory".

Here is the Python code [#1]
import math testList1 = [ [100], [50, 50], [60, 40], [70, 30], [80, 20], [90, 10], [25, 25, 25, 25], [40, 30, 20, 10], [99, 1], ] def entropyGet1(list2): sum1 = sum(list2) e1 = 0.0 for x1 in list2: x2 = x1 / sum1 if x2 != 0.0: e1 += x2 * math.log(x2,2) return - e1 def entropyDo1(list1): print("Entropy Distribution") print("") for list2 in list1: e1 = entropyGet1(list2) print("{0:5.2f} {1:s}".format(e1,str(list2))) entropyDo1(testList1)
Here is the output of the Python code.
Entropy Distribution -0.00 [100] 1.00 [50, 50] 0.97 [60, 40] 0.88 [70, 30] 0.72 [80, 20] 0.47 [90, 10] 2.00 [25, 25, 25, 25] 1.85 [40, 30, 20, 10] 0.08 [99, 1]

Languages vary in how many letters they contain in making words.

• The Greek alphabet has 24 letters.
αβγδεζηθικλμνξοπρστυφχψ
• The English alphabet has 26 letters.
ABCDEFGHIJKLMNOPQRSTUVWXYZ
• The Russian alphabet has 33 letters
АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ

The average frequency with which letters happen in words (in a given corpus of texts) can be determined and varies from letter to letter.

If each letter appeared with the same frequency, then, for English, the entropy would be as follows.

Does this make sense?

So the log, base 2, of 26 is between 4 and 5 bits.

Note that all logarithms are related by a constant factor. Thus, the following are true.

Here is the Python code [#2]
import math x1 = 26 e1 = math.log(x1)/math.log(2) e2 = math.log(x1,10)/math.log(2,10) e3 = math.log(x1,2) print("Logs of {0:d}:".format(x1)) print("") print("\t{0:0.3f} (default base - 10, adjusted for base 2)".format(e1)) print("\t{0:0.3f} (base 10, adjusted for base 2)".format(e2)) print("\t{0:0.3f} (base 2 directly)".format(e3))
Here is the output of the Python code.
Logs of 26: 4.700 (default base - 10, adjusted for base 2) 4.700 (base 10, adjusted for base 2) 4.700 (base 2 directly)

On average, guessing a letter in a puzzle, without any other information, requires 13 guesses (for 26 letters).

How many guesses would it take to fill in the missing letters in the following?

Some languages, such as Hebrew, do not have vowels, only consonants. One learns to infer the vowels according to the context in which the consonants appear.

• The Hebrew alphabet has 22 letters.


אבגדהוזחטיכלמנסעפצקרשת

Shannon showed that the English language has an entropy of between 0.6 and 1.3 bits.

The redundancy of natural language helps in removing ambiguity when not all of the text can be transmitted accurately, as in listening to a broadcast with a lot of static noise.

Google has collected a large amount of data on letter sequences in the text of various languages.

Such "grams" are known as "bi-grams" (2 letters) , "tri-grams" (3 letters), and so on. The general term is "n-grams" where n is an integer (greater than one).

The bit length of a security key, such as 1024 bit, is not always a maximum entropy security key.

You need the max entropy value to compare security by the bit length of keys.

A true random number generator needs a lot of entropy.

The GRC web site, by Steve Gibson (famous security pod-caster), provides passwords that have high entropy.

Every one is completely random (maximum entropy) without any pattern, and the cryptographically-strong pseudo random number generator we use guarantees that no similar strings will ever be produced again.

See https://www.grc.com/passwords.htm. There are more details there on how maximum entropy is used/achieved.

Pseudo-random numbers appear random but actually have a pattern. True random numbers cannot be generated via strictly computer methods.

Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin. John Von Neumann

From WikiQuote:

In mathematics you don't understand things. You just get used to them.

If people do not believe that mathematics is simple, it is only because they do not realize how complicated life is.

There probably is a God. Many things are easier to explain if there is than if there isn't.

From WikiQuote:

He was a really remarkable man. He listened to me talk about this rather obscure subject and in ten minutes he knew more about it than I did. He was extremely quick. David Blackwell

Applications:

• Security: random numbers required (assumed intelligent adversary - humans)
• Simulation: pseudo-random numbers work well (non-intelligent adversary - nature)

Note: Raspberry Pi has hardware random number generator. How well does it work?

The concept of genetic entropy is the tendency of the coded information in genetic material to accumulate mutations and degenerate over time due to errors in the copying process.