A customer in the labor time-keeping business had a series of timeclock products that were installed on factory floors to allow the workers to clock in and out, and to change jobs they were working on.
The first versions of these clocks were connected via serial lines, and in 1992 I was contracted to write the UNIX server software that would communicate with all of them. A configuration file gave the device names of all the remote clocks, plus some key parameters, and it would poll them in turn periodically. When any clock had data, it would be saved into a logfile that was later passed on to other software for processing.
This software was in production for many years, and was eventually ported to several other variants of UNIX. Though my software implemented the serial protocol, I did not design it.
In 1995, my customer migrated to network-based timekeeping, and the clocks they used ran MS-DOS and used a third-party TCP/IP stack. These units had a much richer feature set than the previous RS232-based clocks, and I was contracted to develop the protocol, the libraries that talked to the network, and the server component.
At the time, this third-party stack was expensive - several hundred dollars per copy - and we seriously considered implementing our own stack to save these substantial costs. Doing a full TCP/IP stack from scratch is no small project, but since we designed our protocol to use UDP only, the elimination of TCP would have eliminated an enormous amount of complexity. Ultimately we never did write our own stack, but the UDP-based implementation remained nevertheless.
"Application" packets are simply passed up from the library to the client or server for application-specific processing, and the library does nothing with them save for validating the checksum and silently dropping duplicate responses.
"Transport" packets are used for library-to-library communications, and this includes server discovery, file transfer, and other debugging/management commands. The application layer itself won't ever see these packets as they are processed entirely inside the library.
The library handled the entire transaction asynchronously, and the application could either wait for completion, or continue processing until the transfer-complete callback was invoked.
The servers were ported to multiple UNIX systems, and remained in production for many years. The packet structure I designed initially required no changes over the life of the product.