Data in many applications, such as data science applications, are often created, distributed, and used in a compressed file format containing many folders and files using the ZIP file format.

To compress something is to make it take up less space or time.

• Less space is needed to store it.
• Less time is needed to move it.

Can we compress a car? We can compress a car and it will take less space. But it is not easy to recover the original car.

We can compress data and then recover the data.

Some students would like it if the class were compressed so that they could get out early.

Internet downloads are quicker using compression. Here is the method.

• Compress the file.
• Download the now smaller file.
• Decompress the downloaded file.

It does not work well to download a pizza!

Compressed files are sometimes called zipped files, a term derived from the PKZip software program. That is, you unzip a zipped file to decompress it.

Operating systems have built-in decompression software for zip files.

There are two types of compression.

• lossless compression
• lossy compression

In lossless compression, nothing is lost. Examples:

• programs
• data files

Image examples: *.bmp files, *.gif files.

In lossy compression, some bits are lost. Examples:

• images
• sound

Image examples: *.jpg files, *.wav (although the uncompressed wav format can be easier to work with in some applications).

Zip files are a way to combine and compress multiple folders and files into one file.

• Zip the folders and files to a zip file.
• Unzip the zip file to the folders and files.

A zip file is unrelated to a USPS (United States Postal Service) ZIP (Zone Improvement Plan).

The popular North American term zipper, (UK zip, or occasionally zip-fastener), came from the B. F. Goodrich Company in 1923. The company opted to use Gideon Sundback's fastener on a new type of rubber boots (or galoshes) and referred to it as the zipper, and the name stuck. Wikipedia

Mozilla applications such as Firefox (browser) and Thunderbird (email client) store their add-ins in the form of ZIP files with extension xpi for XPI (Cross Platform Installer).

An XPI (XPInstall) file is simply a ZIP file containing the extension files, with the install.rdf file at the root of the ZIP. Users can download and install XPI files off a website or from a local file, by opening it or dragging it into the Extension Manager window. https://developer.mozilla.org/en/Extension_Packaging (as of 2010-02-01)

Files in OpenXML can be the source of textual data for data science.

OpenXML is an open XML standard for representing textual data in the form of documents, spreadsheets, presentations, etc. The old proprietary binary file format is in parentheses.

• docx - documents (doc)
• xlsx - spreadsheets (xls)
• pptx - presentations (ppt)

Each file is a collection of folders and files in ZIP file format.

The following files are renamed zip files.

• apk - Android package
• jar - Java ARchrive
• cab - Cabinet file (Windows folders and files)

A traditional file system contains folders and files. One navigates the tree-structured directories of folders to organize and find files.

An SQL (Structured Query Language) database is a relational database designed to minimize redundancy of data using relations and normal forms.

A NoSQL (originally referring to "non SQL" or "non relational")[1] database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases. Wikipedia

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster. (Redis web site) Redis (Remote Dictionary Server) is a form of NoSQL database.

Redis popularized the idea of a system that can be considered at the same time a store and a cache, using a design where data is always modified and read from the main computer memory, but also stored on disk in a format that is unsuitable for random access of data, but only to reconstruct the data back in memory once the system restarts. (Wikipedia)

Redis is a key-value in-memory database often used to improve web performance. I have used Redis to improve network performance of distributed processing of problems (e.g., legal work, topic modeling, etc.) where 10 to 20 or more machines work on the same problem at the same time (needed for the legal work, can be scaled up using small credit-sized computer boards).

AWS (Amazon Web Services) provides cloud services and storage in NoSQL form.

In such a form, what appear to be folders and files are just files where the folder separator, such as "\" or "/" are just characters in a name. The navigation provided is illusory and one needs to be aware of that. Complex pattern matching (regular expressions, etc.) are used to find and search for files, etc.

Relational database concepts and thinking are still valuable, but not directly supported. One needs to be aware of this.

The format for folders and files in a ZIP file are similar to NoSQL in that what appears to be a folder and file structure are illusory, It is a flat file system that appears to be hierarchical.

Packages provide ways to think about the folders and files in a hierarchical manner. One needs to be aware of this when programming ZIP file adds and reads.

In a zip file, only files are stored. Folders are just a prefix of text for a file. In some cases, tricks or special handling is needed to include folders in which there is no content.

A zip file can contain compressed files. If the compression ratio is large, there is a lot of redundancy in the file.

Python can be used to both create and read zip files.

Python can read most zip files with password encryption but cannot (without third party support) create zip files with encryption.

In most cases, calling an external program such as 7-Zip can provide a way to create zip files with encrypted password.

Python can be used to read a zip file.

The zipfile package is required (installed as part of Python).

In a previous Python example, a docx was created. This docx file is read and the folders and files displayed. Here is the Python code [#1]
import zipfile docxFs1 = "./doc1.docx" print("Docx file: {0:s}".format(docxFs1)) zip1 = zipfile.ZipFile(docxFs1) for info1 in zip1.infolist(): print("\t{0:s} [{1:,} from {2:,} bytes]".format(info1.filename,info1.compress_size,info1.file_size))
Here is the output of the Python code.
Docx file: ./doc1.docx [Content_Types].xml [411 from 1,788 bytes] _rels/.rels [248 from 734 bytes] docProps/core.xml [361 from 721 bytes] docProps/app.xml [491 from 1,132 bytes] word/document.xml [1,706 from 6,279 bytes] word/_rels/document.xml.rels [325 from 1,359 bytes] word/styles.xml [12,147 from 349,458 bytes] word/stylesWithEffects.xml [13,625 from 438,131 bytes] word/settings.xml [959 from 2,535 bytes] word/webSettings.xml [256 from 438 bytes] word/fontTable.xml [611 from 2,811 bytes] word/theme/theme1.xml [1,734 from 10,939 bytes] customXml/item1.xml [167 from 262 bytes] customXml/_rels/item1.xml.rels [189 from 295 bytes] customXml/itemProps1.xml [225 from 354 bytes] word/numbering.xml [875 from 5,513 bytes] word/media/image1.png [8,839 from 12,204 bytes] docProps/thumbnail.jpeg [1,469 from 8,324 bytes]

Note that the amount of compression varies from file to file.

A large amount of compression means that the original file had a lot of redundancy in it.

Note: The OpenXML file format is very complex in that there are many internal references and cross-references that need to be exact.

Example: In PowerPoint if any reference or cross-reference is not correct, the Presentation will be corrupted and may not load at all. These formats are not something that one would manually create.

Python can be used to create a zip file.

For compression, the zlib package is needed (installed as part of Python).

The following program then opens and reads the zip file just created. Instead of dynamically creating a file and then adding it, the writestr method is used to write text directly (as if it were a file). Here is the Python code [#2]
import zipfile import zlib textList1 = [ "one", "two", "three", "four", "five", ] zipFs1 = "./myzip1.zip" print("Zip file: {0:s} (add)".format(zipFs1)) zip1 = zipfile.ZipFile(zipFs1,mode="w",compression=zipfile.ZIP_DEFLATED) try: for text1 in textList1: zipName1 = "DATA/{0:s}.txt".format(text1) print("\t{0:s} as \"{1:s}\"".format(zipName1,text1)) # write an existing file: # zip1.write(textFs1,zipName1) # write text directly: different arg order zip1.writestr(zipName1,text1) finally: zip1.close() print("Zip file: {0:s} (read)".format(zipFs1)) zip1 = zipfile.ZipFile(zipFs1) for info1 in zip1.infolist(): z1 = zip1.open(info1.filename) bytes1 = z1.read() z1.close() # directly convert bytes to text text1 = bytes1.decode("UTF-8") # or one can write the binary bytes to a file using mode wb # then open the file as text and read as text print("\t{0:s} as \"{1:s}\" [{2:,} from {3:,} bytes]".format(info1.filename,text1,info1.compress_size,info1.file_size))
Here is the output of the Python code.
Zip file: ./myzip1.zip (add) DATA/one.txt as "one" DATA/two.txt as "two" DATA/three.txt as "three" DATA/four.txt as "four" DATA/five.txt as "five" Zip file: ./myzip1.zip (read) DATA/one.txt as "one" [5 from 3 bytes] DATA/two.txt as "two" [5 from 3 bytes] DATA/three.txt as "three" [7 from 5 bytes] DATA/four.txt as "four" [6 from 4 bytes] DATA/five.txt as "five" [6 from 4 bytes]

Here is how the created zip file appears in WinZip with the "DATA" folder expanded.

Note: For very small amounts of text, compression may actually increase the size of the needed bytes.

Note: There are many other options available for time and date stamps, comments, etc.