[流れ]

Data flows through space

Systems revel in balance

The enmeshed gates sing

• Core Principles of System Design
  • Simplicity: Minimize Complexity
  • Performance: Balance Efficiency \& Resources
  • Predictability: Consistent Behavior Over Time
  • Correctness: Accuracy \& Precision
• Data Handling Strategies
  • Data Locality and Movement
  • Data Shapes and Structure
  • Pre-Processing for Efficiency
  • Composable Across systems
  • Data Hierarchy and Latency Reduction Scalability
• Scalability \& Performance Optimization
  • System Bottlenecks and Design Avoidance
  • Cost Awareness: Balancing Tradeoffs
• Adaptability and Usability
  • Layers of Levers: Configuration
  • Graceful Failure: Handling Errors
  • Documentation \& User Experience
• Conclusion: Building Elegant Systems
• Appendix I - Case Studies \& Practical Examples
  • Factorio’s Pathfinding Algorithm
  • Zip-64 and Tape Storage Limitations
  • Rainbow Tables in Cryptoanalysis
• Appendix II - Links to More
  • Core Principles
  • Data Handling
  • Adaptability and Usability

Core Principles of System Design

Simplicity: Minimize Complexity

Avoid unnecessary features, but balance the minimalism with practicality. Systems should have a narrow scope of problems to solve. Feature and scope creep pull attention away from the central purpose of a system and introduce additional “error-surface” for bugs and issues to grow from.

Minimize the amount of stateful components in the system. If you’re storing any kind of information for any amount of time, you have a lot of tricky decisions to make about how you save, store and serve it. Stateless systems are simpler to test and verify. By pushing stateful to the edges of the system, it removes complexity from

The design should minimize edge cases in applied logic. This makes it easier to mentally model the system and reason about functionality.

Performance: Balance Efficiency & Resources

Adding more resources improves performance of a system, to a point. The various limits of hardware and software all scale differently from each other, so performance characteristics end up differing across scales of operation. Parallelism, pipelining, and serialization are common design patterns to spread work across resources. The scaling structure used should be derived from the structure and behaviors of the input data.

While compute and memory are usually the primary focus, all aspects of the system should be considered. Financial, network, disk, data speed/latency and power usage are all considerations in improving efficiency.

For best performance the system should exist in a balance of input and processing resources. There should exist slack in the system, relational to the volume and processing cost of the system input. If resources are saturated and there is a surge in input, there will be an greater impact to performance than if there are unused resources able to accommodate the additional load. In my experience across computing applications 80-85% utilization is around the sweet spot.

Predictability: Consistent Behavior Over Time

Best / Average / Worst performance of the algorithm should be similar over the range of possible inputs. To illustrate, sorting algorithms are simple to reason about.

Quick sort first partitions the array and then make two recursive calls. Merge sort first makes recursive calls for the two halves, and then merges the two sorted halves.

They are roughly comparable in the best and average case, there is a drastic difference on worst case performance. Algorithms may also perform differently on different scales of input data.

Despite this, a program could utilize both algorithms by detecting properties of the data before processing and using the right algorithm for the job.

While processing the data, resource usage should be stable. There shouldn’t be choppy CPU usage, and Expensive operations like memory allocation and disk/network requests should be performed up front. They should make data available to the core algorithm without hoarding CPU.

Correctness: Accuracy & Precision

Algorithms should produce accurate results under all valid inputs. They should adhere to specific constraints such as idempotency. For simple problems math, a calculator would be useless if it performed carries on a portion of the time. Problems modeled by systems need to have measurable outcomes.

Outputs derived from data should be factually correct (see LLMs) Processing the same set of data twice should result in the same output.

In sorting algorithms, there is the concept of stable vs unstable sorting algorithms. A stable algorithm will preserve the relative order of equal elements, while an unstable one may or may not shuffle the order of equal elements. While this may not matter for sorting raw values, it can be valuable when sorting data structures on a field. More entropy is introduced into the system but it is up to the goals of the system to determine if it’s relevant or not.

A concept from physical sciences is systematic error vs. random error. Systematic error is a consistent, reproducible error that stem from within the system. This error can be:

• Clock drift
• Floating-point arithmetic
• Resource contention
• Outright error (2+2=5)
• Unchecked expiration

Random error causes include instrument limitations, minor variations in measuring techniques, and environmental factors.

• Input issues/fidelity
• Security
• Upstream outages
• Weather events
• Interstellar rays
• System just didn’t feel like it

Systematic errors can mostly be solved. Because they arise from the architecture and design of the system they can more directly be confronted. Random errors can be mitigated but not eradicated. The system exists within reality, but steps can be taken to minimize and compartmentalize the error. “Swiss-hole” security allows individual layers to have failures/issues that can be covered by other layers. Of course, if the holes line up you’re gonna have a bad time.

LLMs take an interesting approach by often being incorrect and confident. From a high level, they are deterministic. Provided the same seed and a 0 temperature, any model should provide What is not deterministic is the system operating the LLM. Differences in hardware, scheduling, and parallelism can shuffle the order of operations in a non-deterministic (from our perspective) way.

Data Handling Strategies

Data Locality and Movement

Algorithm should be able to perform tasks on a subset of the data. It should not require processing all the data to get a result (processing everything required for the result).

Data should be stored in a way that’s conducive to how it will be processed. The algorithms data access patterns should integrate cleanly with how the data stored. Usually, algorithms fall into either being message or stream oriented. Data stored with random access make it easier to batch and compartmentalize work.

Algorithms that require random data access do NOT perform well with stream-oriented data sets

Memory access patterns should be optimized to minimize cache misses and maximize reuse of data in memory. Individual, discrete units of data should be processed as few times as possible, extracting what’s needed for the lifetime of the algorithm in minimal passes.

At a programming level, this could mean utilizing hash maps and dicts to group data instead of using un-indexed lists. Data storage-wise, this could mean s3 buckets with data spread across a folder tree that groups files on some valuable property.

Data Shapes and Structure

What data shapes and structures are consumed and produced

Data structures describe how data, metadata, and references are contained.

The organization of data to play to strengths of algorithm. Don’t want to over-index, or chop up data into too-fine pieces. Analogy to min-maxing surface area and volume of a square. Indexing shouldn’t be overly specific. Meaningful clustering of data.

Graph traversal is a core computing problem where storage structure can drastically reduce work. With just a raw list of nodes and edges, it can be expensive to determine paths between two nodes.

By storing the graph in a hash map, nodes can readily be hopped between. The data will be able to be explored as a graph by hash lookups rather than iterating through a list.

Having the raw data organized in a map is nice, but additional information about the data can be gathered to further improve task performance. “Regularization” of the input data can help simplify the problem you’re trying to solve.

Structure decisions control the way in which data is processed and interpreted.

Pre-Processing for Efficiency

https://en.wikipedia.org/wiki/Predictor%E2%80%93corrector_method

Pre-processing data before it enters parts of the subsystem. Key information about the data can be extracted before the primary processing takes place to better inform and optimize the processing.

For example, the speed of a search algorithm on a list will vary wildly depending on if the list is sorted or not. Over an unsorted list, an algorithm would need to analyze each list item on it’s own. By sorting the list beforehand, the algorithm can rely on guarantees inherent in the data being processed and utilize better algorithms.

The pre-processing can take place during or outside of the algorithm’s execution lifecycle. Lookup tables are a common example of using pre-computing to improve task performance. Before computers there were used to speed up calculation of complex functions in trigonometry, logarithms, statistics and more.

Composable Across systems

Instead of solving every problem, it is better to inter-operate with other tools. The algorithm and tool that encases it should readily integrate with other applications and systems.

In the *nix world, most terminal applications are line-of-text oriented. By following the “Unix Philosophy” of managing data as files and text streams, tools like awk, grep, cat, and find can be combined to solve a task.

Data Hierarchy and Latency Reduction Scalability

Performance should be maintained across orders of magnitude of input. The varying data sizes should be able to be handled with sacrificing efficiency (linear vs. quadratic growth).

Algorithms should have a clear model for describing how data moves through the system. It should also be aware of the transport of data as it passes through caches, buffers, networks, and physical locations.

Compute should be placed near the input data. Data should be processed near where it is stored to minimize latency side-effects.

In a cloud architecture settings, this looks like cross-AZ and cross-Region data movement. In machine code this could be optimizing data slices for L1 and L2 processor cache. Each layer from the processor doing the work is a magnitude of difference

The more copies of the data that exist makes it easier for them to become out of sync. Cache layers can solve problems, but also create potential for many more. Data Lifetimes

Scalability & Performance Optimization

System Bottlenecks and Design Avoidance

Choosing the right abstraction layer to solve problems.

Avoid system limitations through thoughtful design.

Cost Awareness: Balancing Tradeoffs

Computing costs money. Every layer of abstraction has it’s own costs and benefits.

Our capitalist system uses cost as a driver, the financial aspects are also a driver in resource decisions. With the rise of [SaaS, PaaS, *aaS] it can often be cheaper to outsource functions to achieve “good-enough” functionality in a short timeframe.

The system should optimize for cost per operation within itself. Of course, “cost” needs to be defined by the hopefully benevolent entity managing the system. Time is a large factor, not just of the operations performed but the time-effort cost of constructing, managing and deconstructing the system over it’s lifetime.

Adaptability and Usability

Able to easily adjust to changing inputs and environments.

This doesn’t need to happen “on the fly”, but the code describing the algorithm should be contained enough to be easily applied to other problems.

Many advances in science and engineering have been made by applying an old tool to a new problem. By

Layers of Levers: Configuration

Configuration is an additional way of communicating intent to the system separate from the input data.

grouped by function and breadth of changes. Is meant to abstract away complexities from the user, providing an ergonomic way to “steer the ship” Keep it simple and self-explanatory.

Minimize side affects, provide clear

Graceful Failure: Handling Errors

Errors will always happen. Even if all inputs, states and outputs are accounted for, there are factors outside of the system. Links go down, hardware fails, and cosmic rays love flipping bits. When encountering unrecoverable issues, plan for a graceful landing and exit instead of plowing a burning path of faults, errors, and corruptions.

It is often better to fail out early rather then trying to continue and create bigger issues. In databases, design patterns like transactions and Write-Ahead-Logging (WAL) help minimize erroneous writes from failing clients and servers. By taking steps to preserve state before work is done the system can guarantee (to a degree) the atomicity of actions taken.

User-facing errors should be actionable and explain what went wrong. They should communicate what the system was doing and what input was being operated on. The message should cover the “who what when where” parts of the problem, the “why” and “how” are usually left to the user to solve.

Error codes and the like are valuable as an index, but they require external resources to index against. Most systems aren’t constrained by bits and bytes so messages can be more verbose. In programming I prefer to return errors instead of exception-ing out of the main control flow. Exceptions are expensive, and forces core parts of the system to be ready to “play catch” from potentially disparate sub-systems.

Documentation & User Experience

Algorithms do not operate in a vacuum. They must be cared for and maintained like any living thing. Documentation should outline the use case for the algorithm and how to apply it.

The complexities of the system should be described in an understandable way. It should use ideas and references familiar with the target audience. Having a “system vocabulary” of concepts and modules gives users a scaffold to construct understanding upon.

There should also information on how the system fails. Users should be able to easily determine the cause of errors, and have the tools/knowledge available to solve it themselves.

Even if it is the most optimal algorithm ever constructed for a task, if it is not ergonomic to users than it will be forked or forgotten.

Conclusion: Building Elegant Systems

Everything old is new again. The questions of how to design a computing system have been asked and answered innumerable ways over the decades. Most of the problems we strive to solve are echoes of those faced by our computing progenitors. We as engineers face physical and digital constraints that design is bound to.

“Be vicariously lazy.” - Programming Perl 1991

Appendix I - Case Studies & Practical Examples

Factorio’s Pathfinding Algorithm

Factorio Friday Facts #317 - New pathfinding algorithm describes their process in developing an npc algorithm.

In path-finding algorithms, this can mean grouping nodes by location or travel cost. Group 1x1 locations into say 64x64 tiles. the algorithm can then first process the large tiles to get a rough estimate of a path, then explore more likely tiles before less likely ones resulting in less data processing overall.

Zip-64 and Tape Storage Limitations

Original

The ZIP-64 internal layout places the index of stored files or “General Directory” at the end of the archive structure. This is advantageous for writing a zip file, as the client only knows the data of the General Directory once all files are written. This also can ease adding files, as they can overwrite the old General directory before writing the updated one.

This structure is not helpful when attempting to read an archive. A naive implementation could require processing the entire zip file to read the General Directory and provide it to the user.

Some systems can work around this, but others have hard constraints on how data can be accessed. High-density tape drives provide large amounts of storage, but only read data sequentially. This makes ZIP-64 particularly bad, as the system must first seek to the end of the tape before being able to meaningfully access an archive.

Rainbow Tables in Cryptoanalysis

Original

Rainbow Tables are pre-computed chains of password hashes.

are another, more modern form of lookup tables. They help drastically improve hash cracking performance. Instead of storing every single plaintext and hash, “chains” of hashes are created and only the ends are stored. The target hash is repeatedly hashed using the same method to create the chains. If the result matches one of the chain “ends”, the algorithm can then hash from the head of the chain to determine the input that generates the matching hash.

Appendix II - Links to More

Core Principles

Simplicity

• Simplicity of IRC - Susam Pal
• htmx sucks - Carson Gross
• Great software design looks underwhelming - Sean Goedecke
• The Architecture of Serverless Data Systems - Jack Vanlightly

Performance

• Simulating Time with Square-Root Space - Ryan Williams
• Exploring the Power of Parallelized CPU architectures - meekochii

Predicatability

• Best-Worst-Avg.: Sorting Algorithms
• Quick Sort vs Merge Sort

Correctness

• Stable and Unstable Sorting: Why Stability Matters? - Siddharth Chandra

Data Handling

• Make a Faster Cryptanalytic Time-Memory TradeOff - Philippe Oechslin
• Hash-Based Improvements to A-Priori
• Regularization -
• Using Obscure Graph Theory to solve PL Problems - Sandy Maguire
• Hypergraphs, in the context of index and metadata creation.

Adaptability and Usability

• The Art of README

Website: https://mlcommons.org/working-groups/data/croissant/

TRTL Lang Definition: github.com/mlcommons/croissant/docs/croissant.ttl

The Croissant spec is designed to provide a standard way to describe datasets properties, raw file relationships, and derived dataset attributes.

The Croissant metadata format simplifies how data is used by ML models. It provides a vocabulary for dataset attributes, streamlining how data is loaded […] Croissant enables the interchange of datasets between ML frameworks and beyond, tackling a variety of discoverability, portability & reproducibility, and responsible AI challenges.

For this post I mostly analyzed version 1.0

The Croissant format roughly consists of 3 parts:

A simplified diagram of the Croissant spec

Metadata

Builds on schema.org/Dataset.

The metadata for the dataset is stored in the root of the Croissant file. A URL is required to link to where the dataset can be found. While only one can be set, there is also a sameAs field that allows pointing to matching datasets with different URLs (http, s3, ipfs, onion, etc).

A keywords field allows adding text and links about the dataset, perfect for tags or references.

Versioning and isLiveDataset attributes allow the dataset the change over time. Changes to the dataset should be constrained to updating or creating, not deleting. It is best utilized to add additional data and improve RecordSet information. If the structure of the data is also changing it is probably better to create new, separate datasets.

Resources

Resources refer to the actual files of the dataset. Items are either a FileObject or a FileSet. A FileObject is a single item, and FileSet is one or more items matching a glob pattern.

They are able to reference each other via the containedIn property. This is useful for referencing compressed files like .zip and .tar.gz. A Fileset entry and be created for the contents of a .zip, which is stored as a FileObject. This allows for storing the dataset compressed with the Croissant file guiding individual file extraction.

RecordSets

A RecordSet describes a set of structured records obtained from one or more data sources (typically a file or set of files) and the structure of these records, expressed as a set of fields (e.g., the columns of a table). A RecordSet can represent flat or nested data.

RecordSets do not reference resources directly. Instead, they group Fields together which in turn reference resources. This allows RecordSets to span multiple files, though that definitely introduces additional complexity.

The Field object encodes information about dataset fields such as name, types, and extraction method. They are able to reference other Fields and RecordSets to represent complex data relationships. Each field has a source property that references either a DataSource object or a URL. DataSource link a Field to associated RecordSets and File* resources, acting as a focal point describing what’s required to view a given RecordSet.

The RecordSet-related types are primarily designed for column data, but the referencing allows for describing more complex data structures. An open-ended data field allows encoding data as JSON and reference it from Fields for properties like enums and categories.

Ecosystem Tools

There is of course the spec repository which contains the code for:

• mlcroissant Python library, with cli tool
• web editor, which is live at huggingface.co/spaces/MLCommons/croissant-editor with login
• links to Croissant tools integrations

Conclusion

Most of what I found focused on using Croissant files in data pipelines, not creating them. Dataset management. tools and platforms integrate Croissant as plugins to the system instead of creating distinct Croissant tools. A Croissant file is not the primary dataset management format for applications. Instead, it is positioned as an interchange format for datasets.

Potential changes:

• how to use with homomorphic encryption

Planned changes: https://github.com/orgs/mlcommons/projects/44/views/1

• Make spec encompass the lifetime of the dataset instead of a snapshot.
• defining and using custom “data-level annotations”
• Reference other downstream/upstream datasets
• Add additional dataset structure vocabulary (n-dimensional arrays, geo-spacial, AV)

Applications - The System-User Interface

Programs designed for end-users Focused on specific tasks or providing value to users.

Provide a varied interface to interact with systems. Presentation guides user interaction and places constraints of what the user is able to create.

Key Properties of Applications

• User-centric: Focuses on direct interaction with users through GUIs, APIs, CLIs, etc.
• Task-specific: Often aimed at solving a particular problem
• External Dependencies: Relies on systems such as web browsers, operating systems, databases, etc. to function.
• Modular: Often composed of small components such as plugins or libraries that can be updated or replaced.

Systems - Infrastructure at large

Underlying hardware and software that manages resources, executes code, and enables applications.

Key Properties of Systems

• Resource management: Control processing, memory, disk, network
• Abstraction Layers: Provides APIs or interfaces to simplify interacting with other components
• Foundational: Create the base for applications to run
• Scalable: Designed to handle large-scale operations

Protocols - Calls from Inside the Stack

Rules governing how applications and systems interact with each other. Ensure interoperability, security, and efficiency across networks.

Provide guarantees that applications and systems can rely to communicate information. Span multiple layers of the hardware/software abstraction stack.

Open protocols are more valuable than proprietary ones. Accessability and visibility is important to be able to verify protocol guarantees.

Security properties should be integrated with protocols as a principle. Aspects like checksums provide authentication and integrity guarantees that applications and systems can trust and rely on.

Key properties of Protocols

• Standardization of structure and transport
• Security-focused: Define encryption, authentication, and data integrity mechanisms
• Adaptable: Able to evolve with technological advancements

Interdependencies - Applications, Systems and Protocols Together

In a graph of computing, applications are leaves, systems are nodes and protocols are edges. Application depend on systems to operate. Systems rely on protocols to communicate with applications and other systems. Protocols provide information communication guarantees, and enable applications to function and interact with diverse systems.

• Ender 5 pro hardware w/ direct feed and non-default nozzle bracket
• BIGTREETECH SKR Mini E3 control board
  • Marlin-based firmware; BIGTREETECH build: firmware-ender5-bltouch.bin
• Octopi via Rasperberry Pi Zero W

The new part: BLTouch Probe

NOT wired for Z-homing. Still uses switch triggered by the bed.

Re-Zeroing the Z Axis

G-Code lookup: https://marlinfw.org/meta/gcode/

From M851 - XYZ Probe Offset docs:

… the easiest way to get the Z offset value (assuming your homed Z = 0.0) is to:

• Home the Z axis.
• Raise Z and deploy the probe.
• Move Z down slowly until the probe triggers.
• Take the current Z position (as reported by M119) and negate it. (5.2 => -5.2)
• Set with M851 Z-5.2 and save it with M500.

Get Current settings: M503

Send: M503
Recv: echo:; Linear Units:
Recv: echo:  G21 ;
Recv:  (mm)
Recv: echo:; Temperature Units:
Recv: echo:  M149 C ; Units in Celsius
Recv: echo:; Filament settings (Disabled):
Recv: echo:  M200 S0 D1.75
Recv: echo:; Steps per unit:
Recv: echo:  M92 X80.00 Y80.00 Z800.00 E130.00
Recv: echo:; Max feedrates (units/s):
Recv: echo:  M203 X500.00 Y500.00 Z5.00 E25.00
Recv: echo:; Max Acceleration (units/s2):
Recv: echo:  M201 X500.00 Y500.00 Z100.00 E10000.00
Recv: echo:; Acceleration (units/s2) (P<print-accel> R<retract-accel> T<travel-accel>):
Recv: echo:  M204 P500.00 R500.00 T500.00
Recv: echo:; Advanced (B<min_segment_time_us> S<min_feedrate> T<min_travel_feedrate> J<junc_dev>):
Recv: echo:  M205 B20000.00 S0.00 T0.00 J0.01
Recv: echo:; Home offset:
Recv: echo:  M206 X0.00 Y0.00 Z0.00
# --- Bed leveling coordinate values
Recv: echo:; Auto Bed Leveling:
Recv: echo:  M420 S1 Z0.00 ; Leveling ON
Recv: echo:  G29 W I0 J0 Z0.20875
Recv: echo:  G29 W I1 J0 Z0.16500
Recv: echo:  G29 W I2 J0 Z0.13000
Recv: echo:  G29 W I3 J0 Z0.10375
Recv: echo:  G29 W I4 J0 Z0.16625
Recv: echo:  G29 W I0 J1 Z0.17000
Recv: echo:  G29 W I1 J1 Z0.18375
Recv: echo:  G29 W I2 J1 Z0.13500
Recv: echo:  G29 W I3 J1 Z0.14750
Recv: echo:  G29 W I4 J1 Z0.18375
Recv: echo:  G29 W I0 J2 Z0.16000
Recv: echo:  G29 W I1 J2 Z0.12000
Recv: echo:  G29 W I2 J2 Z0.12750
Recv: echo:  G29 W I3 J2 Z0.09875
Recv: echo:  G29 W I4 J2 Z0.17500
Recv: echo:  G29 W I0 J3 Z0.13750
Recv: echo:  G29 W I1 J3 Z0.15250
Recv: echo:  G29 W I2 J3 Z0.12875
Recv: echo:  G29 W I3 J3 Z0.12000
Recv: echo:  G29 W I4 J3 Z0.14250
Recv: echo:  G29 W I0 J4 Z0.09125
Recv: echo:  G29 W I1 J4 Z0.12125
Recv: echo:  G29 W I2 J4 Z0.12000
Recv: echo:  G29 W I3 J4 Z0.11000
Recv: echo:  G29 W I4 J4 Z0.20125
# ---
Recv: echo:; Material heatup parameters:
Recv: echo:  M145 S0 H185.00 B45.00 F255
Recv: echo:  M145 S1 H240.00 B110.00 F255
Recv: echo:; Hotend PID:
Recv: echo:  M301 P21.73 I1.54 D76.55
Recv: echo:; Bed PID:
Recv: echo:  M304 P41.78 I7.32 D158.93
Recv: echo:; Controller Fan:
Recv: echo:  M710 S255 I0 A1 D60 ; (100% 0%)
# --- Probe Offset from Nozzle 
Recv: echo:; Z-Probe Offset:
Recv: echo:  M851 X-40.00 Y-10.00 Z-3.85 ; (mm)
# ---
Recv: echo:; Stepper driver current:
Recv: echo:  M906 X580 Y580 Z580
Recv: echo:  M906 T0 E650
Recv: 
Recv: echo:; Driver stepping mode:
Recv: echo:  M569 S1 X Y Z
Recv: echo:  M569 S1 T0 E
Recv: echo:; Linear Advance:
Recv: echo:  M900 K0.00
Recv: echo:; Filament load/unload:
Recv: echo:  M603 L350.00 U400.00 ; (mm)
Recv: ok

Adjust Z-Probe Offset: M851

Send: M851 X-40.00 Y-10.00 Z-3.94
Recv: ok

Save Settings: M500

Send: M500
Recv: echo:Settings Stored (685 bytes; crc 53261)
Recv: //action:notification Settings Stored
Recv: ok

Docker Build Caching

One build cache system I was already aware of was Docker layers. A docker image result and build cache can be represented as a singly linked list of layers. Each layer is a dockerfile instruction that is dependent on the layers that come before it. As part of the docker build process, each layer is cached. If the same layer is already cached, it can be reused instead of recreated.

From the Duckduckgo:

• Docker Docs: Build Cache
• Useful.codes: Understanding Image Layers and Caching in Docker

Docker determines if a cached layer is valid by checking:

• layer instruction: FROM, RUN, COPY, etc.
• layer build context
• layers that came before
• build arguments

The layer build context documentation seemed rather vague, with the Context docs stating the context is made up of the files and directories sent to the builder. I was curious

Based on the Docker Image Specification v1.2.0. Each layer is tracked based on it’s DiffID, then the layer diff IDs are stored in the image’s rootfs field, “in order from bottom-most to top-most”:

Layers are referenced by cryptographic hashes of their serialized representation. This is a SHA256 digest over the tar archive used to transport the layer, represented as a hexadecimal encoding of 256 bits, e.g., sha256:a9561eb1b190625c9adb5a9513e72c4dedafc1cb2d4c5236c9a6957ec7dfd5a9. Layers must be packed and unpacked reproducibly to avoid changing the layer ID, for example by using tar-split to save the tar headers. Note that the digest used as the layer ID is taken over an uncompressed version of the tar.

The Docker documentation references using Buildx as the default image building backend, which provides files and input to BuildKit which performs the actual image build. The Docker docs Build Overview and BuildKit cover these topics from a high-level perspective.

I was curious what specific data was processed by Buildx and BuildKit to create the cache layers so I dug into them to see how it was implemented in practice. After getting lost in the buildx code and not seeing any actual caching happening, I found that the actual caching operation was is performed by BuildKit, separate from Buildx. The Buildkit intro post provides an overview of BuildKit’s methodologies early capabilities.

While being built, the image data is converted into a “build definition format” LLB (low-level builder). LLB allows defining complex build definitions through a content-addressable dependency DAG (directed, acyclic graph). The protobuf definition is located at buildkit/solver/pb/ops.proto. As the graph is being created, “branches” can be pruned based on cache validity. To perform the build, the program simply walks the graph visiting all nodes.

// https://github.com/docker/buildx/blob/542bda49f2b25f32fa424c3dc50f5b9b5c50d280/localstate/localstate.go#L23
type State struct {
	// Target is the name of the invoked target (default if empty)
	Target string
	// LocalPath is the absolute path to the context or remote context
	LocalPath string
	// DockerfilePath is the absolute path to the Dockerfile or relative if
	// context is remote
	DockerfilePath string
	// GroupRef is the ref of the state group that this ref belongs to
	GroupRef string `json:",omitempty"`
}

// Stored as GroupRef in State struct
type StateGroup struct {
	// Targets are the targets invoked
	Targets []string `json:",omitempty"`
	// Refs are used to track all the refs that belong to the same group
	Refs []string
}

// https://github.com/docker/buildx/blob/542bda49f2b25f32fa424c3dc50f5b9b5c50d280/commands/bake.go#L526
func saveLocalStateGroup(dockerCli command.Cli, in bakeOptions, targets []string, buildOpts map[string]build.Options) error {
	l, err := localstate.New(confutil.NewConfig(dockerCli))

  // ... checks + metadata provenance mode settings ...

	groupRef := identity.NewID() // <--------- state group identifier
	refs := make([]string, 0, len(buildOpts))
	for k, b := range buildOpts {
		b.Ref = identity.NewID()  // <--------- state group item identifier
		refs = append(refs, b.Ref)
		// ... update build options ...
	}

  // ... checks ...

	return l.SaveGroup(groupRef, localstate.StateGroup{
		Refs:    refs,
		Targets: targets,
	})
}

BuildKit/LLB File Hashing

Each layer of the docker image contains an instruction to execute. A Stack Overflow answer pointed out different instructions have different cache considerations. The aspect I was most interested in was discovering how file hashes were calculated. Modifying a file used in a layer will invalidate the layer, but from experience I’ve found not all properties of a file are considered. For example, after a docker build takes place, you can touch referenced files to update their timestamp without invalidating the layer cache.

Generating context entries for files and directories is performed in buildkit/cache/contenthash/checksum.go is used to calculate the file checksums. Checksum() calls prepareDigest(fp, p string, fi os.FileInfo). Using file info derived from os.LStat(), it passes this to NewFileHash(path string, fi os.FileInfo) hash.Hash.

This function copies parts of the fi param into a fsutil.types.Stat struct which in turn is placed into a archive/tar Header struct by tar.FileInfoHeader().

An interesting aspect of this conversion (and why we can touch files) is that only a portion of the Header fields are set by BuildKit. A hash is then generated from this header and a tarsumHash struct is returned. The header that is hashed is generated by WriteV1TarsumHeader(), which generates a v1TarHeader from file GNU tar Extended File Attributes and v0TarHeader information:

// https://github.com/moby/buildkit/blob/5d47024a21ec128c8caf7af2ab4b83d63d9a4094/cache/contenthash/tarsum.go#L21C1-L36C2
func v0TarHeaderSelect(h *tar.Header) (orderedHeaders [][2]string) {
	return [][2]string{
		{"name", h.Name},
		{"mode", strconv.FormatInt(h.Mode, 10)},
		{"uid", strconv.Itoa(h.Uid)},
		{"gid", strconv.Itoa(h.Gid)},
		{"size", strconv.FormatInt(h.Size, 10)},
		{"mtime", strconv.FormatInt(h.ModTime.UTC().Unix(), 10)},
		{"typeflag", string([]byte{h.Typeflag})},
		{"linkname", h.Linkname},
		{"uname", h.Uname},
		{"gname", h.Gname},
		{"devmajor", strconv.FormatInt(h.Devmajor, 10)},
		{"devminor", strconv.FormatInt(h.Devminor, 10)},
	}
}

The tarsumHash struct is interesting, in that it is a customized hash.Hash implementation. It has a special Reset() function that when called resets the hash state to default, then “primes” the hash object with header properties in the V1 tarsum format.

This code flow is interesting because only the file metadata is processed, not the contents of the file. Only once the header hash been generated then the file contents are hashed.

Docker Takeaways

Docker builds are processes by BuildKit, which creates a DAG of build steps and layers. The resources and properties required to generate the layer, including references to the previous layer, are included in the graph of a docker image build. BuildKit is able to prune subgraphs (layers/work) from the graph by comparing nodes to stored cache entries.

As a result of the build graph caching system, dockerfile often include idiomatic code to better separate layers and place more stable (unchanging) layers earlier in the build process. One example often seen in python and golang images is pre-loading libraries/modules before the rest of the source code. The library references are usually modified less than the code itself, leading better caching and faster image builds.

FROM python:3
WORKDIR /src

# Temporarily use requirements.txt
RUN --mount=type=bind,source=requirements.txt,target=/tmp/requirements.txt \
    pip install --requirement /tmp/requirements.txt

# Now copy over rest of code in a new layer
COPY . .
# continue...

FROM go:1.24
WORKDIR /src

# Download modules first
RUN --mount=type=bind,source=go.mod,target=/src/go.mod \
    # including the go.sum file ensures the downloaded modules match what we expect.
    --mount=type=bind,source=go.sum,target=/src/go.sum \
    go mod download

# Copy project files over and build
COPY . .
RUN go build

As part of the build-level caching, files are a core part of the cache. I found that not all aspects of the file are considered when cached.

Specific to docker’s file caching:

• File contents
• File name, mode, size, type,
• File Modified Time
• uid/gid, uname/gname
• Any additional GNU tar Extended File Attributes (custom name-value pairs associated with file)

To help inform developers on how to best interact with the cache, the docker doc Optimize cache usage in builds exists. It covers a few ways to improve cache performance, including:

• layer ordering- things with more frequent changes should be placed nearer to the end of the dockerfile.
• minimizing context - Use .dockerignore to minimize files included in the build context.
• use bind mounts - minimize extra copy instructions prepping files for later instructions
• Use cache mounts - cache mounts allow adding specific files to the cache. This cache is interesting because unlike the layer cache, it does not depend on previous layers. It is stored outside the build, and sharable across multiple builds.
• Use external cache - By default, the cache is specific to the builder instance being used. An external cache allows defining a remote cache source. This is most often seen in CI/CD pipelines, and allows easy reuse across workflows. More information can be found in the Cache storage backends doc.

RushJS Build Caching

Another build tool I investigated was RushJS, a tool used to building and managing many NPM packages from a single repo. It is part of the Rush Stack family of projects, which is managed by Microsoft. Along with managing dependencies, it uses caching and incremental builds to speed up building projects, especially those with complex dependency chains.

The article Enabling the build cache describes the caching design and methodology. Rush uses two types of strategies for limiting build work:

• Incremental build analyzer that enables skipping projects whose file contents have not changed since the last build. This does not preserve build output.
• “cache restoration”, where tar archives of project build artifacts are restored from cache.

Incremental Builds

incremental builds: https://rushjs.io/pages/advanced/incremental_builds/

The incremental build system allows rush build to skip projects that are up to date. The “update to date” rules are:

1. The project has already been built locally, AND
2. Its input files and NPM dependencies have not changed since then, AND
3. If the project depends on any other Rush projects, those projects are up to date as well, AND
4. The command line parameters haven’t changed. (For example, invoking rush build --production after rush build would require rebuilding.)

It respects and .gitignore files in the project folder Configuration options are stored in [/config/rush-project.json](https://rushjs.io/pages/configs/rush-project_json/)

Changes to input files are determined by file content hashes. All other file metadata seems to be completely ignored. Since modifying a filename would require modifying the files that references it, creating a soft-requirement to have matching filenames.

This feature does not concern itself with build outputs, so if all the inputs match the cache then the rest of the build for that piece is skipped. If the build artifacts are modified outside of the build process they will remain modified even if a build is skipped.

Cache Restoration

The cache storage is file-based and stored separate from source control. It is default-disabled, and is configured by common/config/rush/build-cache.json. Since Rush is designed for large monorepos, the cache is stored in common/temp/build-cache by default. They do note that the folder can be manually placed outside the repository to share between projects. Cache is also able to be centrally stored in a “cloud-hosted storage container” that can be accessed by users. The options are azure-blog-storage, amazon-s3, and http. It seems to be compatible with any storage that speaks aws-s3. The http provider allows using a normal webserver to serve cache objects.

The cache entry is keyed on various project inputs:

• Hashes of source files that are under the project’s folder, ignoring any files excluded by .gitignore
• Hashes of source files under other workspace projects that are dependencies of the project. This applies to cache restoration strategy but not output preservation strategy
• the versions of all external NPM packages that your project depends on, including indirect dependencies
• the Rush command-line parameters used to perform the operation

It also allows for fine-grained configuration of the cache key in the [/config/rush-project.json](https://rushjs.io/pages/configs/rush-project_json/) and [common/config/rush/build-cache.json](https://rushjs.io/pages/configs/build-cache_json/) files. This allows one to specify environment variables and additional files/glob patterns that should be considered. The `cacheEntryNamePattern`is able to configured to also include tokens such as `[os]` or`[arch]`, but must always contain a `[hash]` token.

RushJS Takeaways

Things I liked:

• multi-pronged approach to analyze different parts of the build
  • separate incremental build and output caching actions
• output preservation calculated by the incremental build analyzer, and applied by the build cache.
  • If no input files were modified compared to the previous build, then the project is skipped.
  • Doesn’t concern itself with build artifacts
• cache restoration
  • query cache based on inputs to determine if output files can be replaced by cached+compressed build artifacts
• file-based cache storage
  • separate cache folder that can be shared across projects
  • provides integrations to remote storage options
  • Also able to check in to source control
  • Build cache artifacts stored in compressed tar files
• Cache is keyed on input file hashes, lib versions, build parameters, and optional ENV vars.
• respects .gitignore files

Final Thoughts

A key part of creating cache is properly tracking the build context of input and output artifacts. At a minimum record the hash of file data, but metadata like timestamps can create unneeded cardinality. Files may also need to be minimally processed by the build system to extract libraries and other program references. The context should also includes build environment properties that are able to modify the output of the build, such as environment variables and system architecture.

It can be valuable to also track and store output artifacts, which can be re-used when the inputs allow. User should never be interacting directly with cache files, so file compression can be used to both save disk space and obfuscate files from idle meddling. The storage location for cache files should consider how re-usable a cache entry is, scoping to project, repo, or system.

Users should be provided a few levers to control the cache behavior, such as a build flag to ignore the cache. There should also be ways to filter things from being cached, either through flags, tool config, or existing config (.gitignore).

Further Reading

• CMake Complete Guide Chapter 3 - cache entries keyed to the variable scope
• Avoid Cache Bloat - avoid keeping entries that are unlikely to be referenced
• Azure Caching Guidance - apply cloud architecture principles to software architecture

This is a collection of notes and thoughts on developing and releasing an open source software package. It is based on the process I went through releasing kr8+, a golang tool that processes jsonnet into output files.

Readme.md

The readme is usually the first thing a module consumer sees.

It should provide a simple, succinct summary of what the tools does and who it is meant for. It should funnel the dense information from code and docs into an easy-to-understand format. Writing a readme is an entire process, beyond this article.

Art of README describes the properties of a good readme and goals it should try to achieve. A useful guide/spec that digs into key element is Richart Litt’s standard-readme spec.

Form a high-level, the important sections are:

• Description - what it is, what it does, what it solves
• Background - why it exists
• User Installation and usage - How to install and use
• Documentation links - Where to learn more about functionality
• Developer Setup - How to get the repo setup locally and contribute
• License information - Note licenses used by the code.

Building

Scripting Common Tasks

Any common commands that a developer runs should be committed alongside the code. It helps make sure everyone is running the same tests and commands and (hopefully) getting the same results.

In the case of kr8+, I setup a taskfile to capture setup and common commands. It is similar to a makefile, but a little more verbose and structured as a yaml file.

When documenting these common commands, make sure you capture:

• Dev environment setup - tools, submodules
• Building - code, docs
• Testing - unit / integration tests

kr8+ example

Generating documentation

Writing documentation is difficult because of how quickly it can become out of date. By generating documentation directly from source code it will accurately reflect the current state of the code. It should be treated as an additional artifact of the code, and always generated along with binaries.

Godoc is useful for documenting Golang code. It extracts comments and formats then into a friendly html file.

There are sometimes tool or framework-specific tool that can generate further documentation. In golang, cobra has tools to generate docs for commands. This is great for creating documentation targeted towards end users. Once generated, the documentation should be published in a way that is easy to consume for the intended audience. Tools like MkDocs package up markdown files into a standard website.

Here’s example of generating package docs for kr8+.

Organizing Code

Organize into logical partitions. Group similar functionality together, and Create clear boundaries of ownership within the code and helps reduce mental load when reasoning about functionality.

Separate user-facing code from internal logic.

Extract functions Keep function short and to the point.

Tests

Tests ensure the program is doing what you think it does. They should be created at different points horizontally across functions and vertically across code layers.

Starting from 0 is difficult, Focus on package “root” and “leaf” functions and work your way in.

• Packages and functions that provide core functionality
• User-interaction layer, to compare inputs to expected outputs.

Test outside of the code with integration tests. Make sure the built tool operates as expected on a basic level. Test the interactions users and other tools will make. They can also verify assumptions the code makes like the existence of certain external resources.

This provides a final check to make sure the tool operates as it’s supposed to. In the case of kr8+ it was valuable to generate examples with the tool. It showed how to actually use the tool and ensures the tool is working as expected.

Formatting

Standard formatting makes it easier to read code through a consistent style throughout the codebase. It can also act as a “soft barrier” to change requests, ensuring basic checks have been made.

Automated formatters in editors and part of build process to keep things standard. It frees up the writer to focus on the code instead of properly balancing tabs and braces.

An .editorconfig file can help “maintain consistent coding styles for multiple developers working on the same project across various editors and IDEs”. It is mostly editor-agnostic, though some like vscode require a plugin.

kr8+ example

Linting

Use tools that can be easily configured and ran. Very language specific, For golang there’s golangci-lint which packages up various tools.

I find it is useful to have the linting tool generate a summary output file, and check that into source control. It’s not always reasonable to fix every single issue, especially on a WIP branch. If you create exceptions, make them as specific as possible to avoid swallowing unrelated issues.

Some of the golang ones I found most useful:

• exhaustruct - makes sure struct fields are all initialized. Enabled as needed to check.
• funlen/cyclop/gocognit - help identify complex functions that should be refactored and split up.
• dupl - detect duplicate fragments of code
• gochecknoglobals - minimize global variables
• gosec/govet - find potential security problems

kr8+ example

Attributing/Licenses

Properly attribute sources of code. Call out contributors.

Code references. Add links to references for code. Even once the links rot, they can still provide some context for confusing chunks of code.

Finally, it can be useful to link to other similar projects. good for comparing functionality, and provides easy links in the future to discover ways to improve your own project.

True OSS licenses are preferred.

Releasing

With the code done up with a bow, it’s time to consider making it widely available. Packages can be published in a variety of formats, so choose one that is more ergonomic for your end users.

In the case of kr8+, I opted for a homebrew tap to enable easy installation on Mac and Linux.

Setting up Homebrew Tap

A tap is just a repo with some special files in it. official docs

Homebrew taps do have a special naming convention, where if a repo is hosted on github “we recommend that the repository’s name start with homebrew- so the short brew tap command can be used”. Unless you are making multiple taps, it is easiest to just name the repo homebrew-tap.

Use the brew tap-new command to initialize a starter repo. Update the readme and license as needed. In the case of publishing kr8+, I never interact with the repo directly, instead it is used as a target for GoReleaser builds.

Once setup, users will reference the tap in the format brew install <username>/tap/<formula> (in the case of github>) or the longer brew install <repo url>/<formula>.

kr8+ example

Configuring GoReleaser

GoReleaser “is a release automation tool. It currently supports Go, Rust, Zig, and TypeScript (with Bun and Deno).”a tool that packages and releases Golang code in a variety of output formats.” Documentation for configuring is here.

For kr8+ it was configured to produce:

• Build packages for linux and darwin
• tar.gz archives
• rpm, deb, and apk packages.
• SBOM and Checksums
• Publish package to homebrew

Additionally, automation was setup to only run the full build+publish when a new git tag is added.

kr8+ example

Github Personal Access Token (PAT)

For publishing on github with automation, you should create a Personal Access Token. It can be stored as a repo secret and be consumed by automation to enable publishing release artifacts. In the case of kr8+, it uses GoReleaser to publish the artifacts to a homebrew tap in a separate repo.

To create the fine-grained token, navigate to github.com/settings/personal-access-tokens.

the token will require:

Read access to:

• metadata
• secrets
• variables* environments
• issues

Read and Write access to:

• code
• pull requests
• commit statuses
• actions

Also valuable to limit what repositories the token can access to limit blast radius if things go sideways. When in doubt, it’s often easier to create a new token instead of trying to get an existing one to work properly.

Actions and Workflows Automation

Run same tests and checks a developer would run. Make sure everything checks out, especially after merging into main. In cases of formatting actions, it can be valuable to allow modifying PRs to provide quick fixes to an annoying issue.

Golang pkgsite

Part of the golang ecosystem is part of the golang documentation system. It allows publishing package documentation in a central place. It will parse the project repo and extract documentation just like Godoc. You can learn more on it’s about page.

To add a package, navigate to https://pkg.go.dev/ and you can click a button to have the repo scraped. You can also sync it by using `proxy.golang.org` as a proxy when fetching a module. For example:

GOPROXY=https://proxy.golang.org GO111MODULE=on go install github.com/ice-bergtech/kr8@v0.0.8

kr8+ example

Other Random Things

• Project logo - A image to associate with the project
• Changelog.md - A list of past changes between version - kr8+ example
• Features.md - A list of current project features and functionality - kr8+ example
• Roadmap.md - A list of future planned projects and improvements - kr8+ example
• Project domain - A custom domain to host docs
• Funding - github sponsors, librepay, patreon etc.

Conclusion

Releasing a project includes a lot of things beyond the raw code. There are a lot of supporting pieces that should be in place to best service the end user. When preparing a project it is important to analyze it through different perspectives. You should consider all the types of end users that may view the project and what they find important. The added context and documentation helps users solve their issues faster and hopefully on their own.

Finally, always be improving a project but also know when to move on. It is better to whole-ass one thing instead of half-assing two things.

References

• kr8+
• EditorConfig
• MkDocs
• Kube Documentation Style guide
• Golangci-lint
• GoReleaser
• Using GoReleaser - Carlos Becker
• Homebrew docs

The Open Systems Interconnection (OSI) model is a conceptual model created by ISO that describes how information flows through different layers in a network system. This post describes a similar model that applies to architecting Infrastructure as Code (IaC) at Scale. The layers do not correspond 1-1 to tools and practices used today, but they provide a useful guide for deploying and managing a diverse set of of applications with distinct compute and data requirements.

This is meant to be applied to large deployments, beyond a single cluster or two. The model attempts to encompass most aspects of managing complex deployments of applications and services across multiple environments. A “normal” deployment under this model could have n regions * m stack types * a applications * r replicas * z Disaster Recovery ratio. Beyond the deployment there are also security, regulatory and contractual requirements that influence the design.

By partitioning IaC into layers, we can better understand how different aspects of are configured, monitored, and secured. Each layer will have a difference balance of manual vs. “codified” actions. IaC should aim to be ergonomic and scalable for the operators and maintainers. It should be thought of as weaving together different threads of infrastructure into a cohesive fabric that provides a stable foundation for applications to be built on.

IaC Reference Model

Layer -1: Billing

Someone’s getting paid somewhere. The billing underlies all the layers. To cost to participate in modern “IP Networks” can vary wildly depending on your needs and how you execute. Most of the time this requires a credit card, but there are growing crypto-backed operations that have greater separation from the traditional financial systems. There is always self-hosted intranet options, but there are still raw resource requirements.

This will include things like:

• Cloud provider accounts
• Domain name registry
• ASN/IP Registry
• Self-hosted electricity/internet/hardware

Monitoring / Security

While there are ways to manage these with IaC, it often needs manual intervention and verification. The best automation for this layer is email rules for getting your attention. Core accounts should trigger emails when logged into or updated.

Strong passwords are extremely important. Make them as long as possible, and use a password manager.

You should also take advantage of Whois (now RDAP) privacy and RPKI services. With enough justification, these can be self-hosted but this is not a common approach.

Layer 0: Privilege

The privilege layer manages access to the cloud provider though Identity Access Management (IAM). To use a service, you usually need an account to deploy resources into.

For 1-2 accounts, this can be done manually, but for any more you should require IaC. While changes are made very rarely, changes here effect core pieces of how all other layers operate. It also helps version control the changes for easy reference and rollback.

Constructing a tree of least-privilege.

Layer 1: Cluster Network

This network is used for private communication between nodes and resources. For very simple deployments this may be contained within a single node or VPC. For larger deployments, this will encompass multiple nodes, networks, and cloud providers.

For anything more complex than a single VPC you will need to start using tunnels to create your network. Traditionally IPSec tunnels were used, but more recently Wireguard tunnels have become popular for being simpler and more efficient.

• Tailscale
• Netmaker
• Aviatrix
• Raw Wireguard

Structure

Separate “tiers” of cluster networks statically define network with varying levels of internal and external access. Resources in the other layers should be able to reference these tiers for baseline configuration. The tiers should help reinforce secure resource isolation to limit attack surface.

• Core - Command and Control resources, including internal-facing Application Networks
• Internal - Applications and client services
• External - DMZ-type tier, containing external-facing Application Network pieces

Layer 2: Permissions

Setup authentication and operational access to cloud resources. This set of access permissions allow API access for yourself and automation. It is distinct from the operational access permissions in Layer 1 in that this layer focuses on operational access rather than management access.

This encompasses managing authentication and authorization for your cloud resources and applications. It usually ends up being a conglomeration of a few different tools that manage different pieces of that auth like SSH keys, SSO, and IAM roles.

Centralize management of authentication, authorization, accounting.

Layer 3: Compute

Raw instances and services that run your applications. Holistically, this can be anything that transforms an input to output but practically it’s an internet-connected CPU with some amount of usable RAM.

• Physical Nodes - Servers, Raspberry Pis, Android phones
• Virtual Machines - EC2 Instances, Droplets, VMs
• Cluster-as a service - EKS, GKE, AKS
• Serverless - Lambda, FaaS

Broadly organize work to minimize resources needed to complete task, but don’t over-optimize. For clusters, consider consolidate or split up nodes to keep average utilization high while also absorbing spikes.

With modern cloud computing, the bottleneck is often memory but this is directly tied to workloads being ran. With enough integration of information about the infrastructure, you can optimize for $/mem, $/cpu, or $/egress.

Layer 4: Storage

Applications often need to persist data beyond the lifetime of the program. Each one has it’s own special way and practices, but they can be roughly grouped into:

• Block/File/Object storage (cluster-local or managed)
• Databases
• Backup+Restore process
• Caching

Data locality is also a key concern for performance and security. Data locality is the practice of keeping data close to where it’s needed, reducing latency and improving efficiency. Storage resources are located:

1. Node/cluster local - Share compute with application
2. Cluster cluster - Clusters managed by you, backed by raw provider
3. Managed service - DB-as-a-Service

Keeping data close to where it’s needed also helps reduce costs, as you aren’t burning egress fees shipping bits around needlessly.

Persistence

Data is large, arbitrary, and often unstructured. Data should be stored in a way that is scalable, durable, and secure. It should also attempt to minimize duplication. Filesystems like ZFS or Btrfs can be configured to deduplicate and compress data at the block level. Backups should also follow the incremental “snapshot” paradigm, with full backups being taken at regular intervals.

When storing the data long-term, 3-2-1 principle should be followed:

• 3 copies of the data
• across 2 mediums
• with 1 copy off site

This ensures data is available even if one of the mediums fails. Of course in the event of some incident, the issue becomes is restoring data in a timely predictable manner.

Layer 5: Application Network

This layer manages application-external access. selectively links parts of the application available to external networks. You “target audience” may be the wider internet, internal clients/teams, or other cluster applications.

This is separate from Cluster Network, in that it’s focused on the application itself, rather than supporting the core communication of the cluster as a whole. These pieces are usually deployed alongside the application resources they are supporting.

Here are some examples of components in this layer:

• Application DNS - Usually A and AAAA records, but includes MX, TXT and others
• Load balancers - Distributed balancers to route traffic to backend nodes
• Ingress gateways - Manages routing rules at L3(Layer 3) and L4(TCP/UDP/etc.) + L7(HTTP(S)/DNS)
• Proxy / Firewalls / Filtering Policy - Manages traffic flow and security at the application level

Layer 6: Application

These are the actual applications that run on the infrastructure. There is an extreme variety of deployment methods, so it is usually best to focus on a way that is ergonomic for both the application and maintainer.

The application itself is beyond the purview of this article, but it should aim to be encapsulated in a easy-to-consume manner. Docker containers seem to have become the most popular, but can be overkill depending on the purpose of the application deployment. Whatever solution used, as much of the configuration as possible should be statically defined.

Documentation needs to be included. If something isn’t documented, it doesn’t exist. The same configuration used to prep and deploy the application can be used to provide greater documentation context.

Layer 6.5: Inter-Applications Integrations

With multiple applications comes the interaction of applications with each other. This layer encodes the services and resources an application makes available, and how they can be consumed. The configuration is highly application-dependent and involves creating “glue” configuration between applications, often through a service mesh or API gateway.

Resources in this layer can encroach deeply into the applications and infrastructure, making them difficult to manage. The configuration may also end up located with the application UI, and not easily accessible via IaC config.

If there is a lot of investment in this area of configuration, it may be worth considering “pushing down” the configuration into lower layers so that it can be more easily managed and maintained.

Architectural Design Considerations

Security

Each layer has it’s own security considerations. Typically for an application to be at all usable there needs to be some sort of information exchanged two it. The challenge is balancing usability and security.

Management of the infrastructures security should follow Tenants of Zero Trust:

• Data and compute are resources: The raw data and compute are treated the same as application from a security perspective. Properties like region and provider are aspects to be considered.
• All communication is secured: All communication between data and compute should utilize secured communication at multiple layers with Wireguard tunnels, network policy, https etc.
• Per-resource, per-session access: The trust boundary should be as close to the application as possible.
• Determine access through collection of client and env properties: Raw credentials should obviously be considered, but additional bits of information can be used to make a more informed determination like source IP or session event timings.
• Monitor the security: Red team yourself to constantly verify the protections in place are operational and effective. In that vein, security pieces that do have issues should most often fail closed.
• Security is a cycle of scanning, analysis, adjustment: Threats are constantly evolving, and your security needs to as well. Patching should automated to the point of approval to make it as frictionless as possible.
• Use system state information to improve security posture: At a minimum all auth should be logged. Collect what you will use. While it may be inviting to mirror, decrypt and analyze complete netflow data, that quickly becomes cost-prohibitive. There are sharp diminishing returns, so the volume should be calibrated to your security posture.

Monitoring

Unless there are regulatory or contract requirements, “lossy logging” can be an effective way to extract value from logs while minimizing the volume processed and stored. High volume logging close to the source and pass on less information as the logs are transferred within the system. As that data is replicated further from the source, funneling the data is important. The amount of log traffic can be funneled/reduced by various means:

• Minimize logging: Who needs debug? Make it easy to enable, but if no ones looking then limiting logging to warning and above reduces log chatter
• Deduplication - Capture logs in a window only pass on changes in values.
• Summarization - Collect sets of logs and perform light
• Sampling - pass on a percentage of logs, simply dropping anything thats not passed on. This is under tha assumption that the logs are not critical and can be lost without consequence.

Source of Truth

Idempotent definitions. Take advantage and templating and configuration management to minimize the amount of manual intervention required.

There are innumerable methods to organize code, whether it a mono-repo, component repos, or monoliths. The key is to choose one that fits your organization’s needs and stick with it. What’s really important is that it is consistent and repeatable.

Configuration should be derived from the applications needs in a standard way to minimize snowflakes. Code should be built on references to standard pieces instead of copies to minimize drift and decay.

Shared, application-level configurations should additive, allowing resources to selectively “opt-in” to sets of properties instead of being assigned a single, restrictive application type. By defining the properties as sets, features from set theory algebra can be applied.

Lifecycle

All applications and resources will have a lifecycle scale and rate. Like documentation, any resource deployed is almost immediately out of date. Resources are in a constant state of flux making IaC a dynamic and evolving process.

While it is impossible to capture all aspects of a system as code, getting things “good enough” is good enough. By virtue of being code, it can also be updated and improved over time. Writing (describing?) resources is no different than writing regular business-logic-type code, there are just a lot more moving pieces.

• Keep it organized - flatten abstractions behind configurable, sensible defaults
• Keep it simple - Minimize complex dependencies that cross resource boundaries.
• Use tools available - linting, perf analysis, and testing.
• Design for scale - Consider how the system will grow and adapt to change.

Conclusion

The easy part is deploying the service. The application developer environment has changed greatly since even just a decade ago. There are many more tools, libraries, and frameworks available to developers leading to more standard, predictable applications. The advent of broader generative AI has made it easier to write code, but also more difficult to reason about the system as a whole.

Architecting the platform and integrations around the service is the real challenge. The platform is like any other product, and needs to provide guarantees to the applications running on it. It does not operate in a vacuum, it is a living system that needs to be maintained and adapted over time. Like growing a tree, it takes care and patience to create a solid system. Unlike a tree, everything is moldable, replicate-able, and roll-back-able (mostly). By keeping the lifecycle of the resource in mind you can better anticipate and negate issues that may arise.

Further Reading

• Lee Briggs - Structuring IaC
• Nathan Peck - Rethinking Infrastructure as Code from Scratch
• OWASP - Cloud Architecture Cheatsheet
• Hacker News Comments

While Proxmox itself does not support cloud-init out-of-the-box, there are ways to integrate it with your Proxmox environment. There are some extra steps to configure a QEMU image format².

On you proxmox host, you will first need to install libguestfs-tools and the image you want to modify and use. In this example, I get debian 12 generic cloud image.

apt install libguestfs-tools
mkdir -p /var/lib/vz/template/qemu &&  cd /var/lib/vz/template/qemu
wget https://cloud.debian.org/images/cloud/bookworm/20250316-2053//debian-12-genericcloud-amd64-20250316-2053.qcow2

Now that the image is downloaded, we need to patch it with virt-edit to add the cloud-init files.

virt-edit debian-12-genericcloud-amd64-20250316-2053.qcow2  /etc/cloud/cloud.cfg
virt-edit debian-12-genericcloud-amd64-20250316-2053.qcow2  /etc/ssh/sshd_config
virt-edit debian-12-genericcloud-amd64-20250316-2053.qcow2  /etc/motd

Now we will import the raw image into proxmox, and mark it as having cloud-init support:

qm importdisk 100 ./debian-12-genericcloud-amd64-20250316-2053.qcow2 local-lvm
qm set 100 --ide2 local-lvm:cloudinit

ternary computing

Ternary computing uses 3 states instead of 2. A ternary system is able to store information more densely than a binary due to its increased state capacity. This density allows for more efficient data handling and potentially higher performance in specific applications compared to binary systems. However, it also introduces new challenges such as the need for specialized hardware and algorithms designed to operate within the ternary framework.

Ternary logic has performance benefits for certain types of computations like matrix multiplication.

Balanced ternary computing uses three states (-1, 0, +1) instead of two (0, 1). Unbalanced ternary (0, 1, 2) could also be applied to some applications. These systems allow a value to be represented multiple ways, e.g. 3 can be represented as (1, 0, -1) or (0, 1, 1). The encoding of the value allows certain computations to be performed more efficiently

Modern ternary-native hardware: https://www.ternary-computing.com/history/CPU-History.html

ternary llms

1-bit llms show better efficiency in terms of speed and energy use compared the usual fp16 type format. 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs .

1-bit model architectures can very cleanly be mapped onto ternary logic, which can in turn be implemented using FPGAs.

To integrate with existing binary-native hardware, a hybrid approach could be used where the ternary logic is implemented on FPGAs using 2 bits that provide the normal (-1, 0, +1) values along with a hardware “null” or special purpose flag value.

architecting hardware to data

Goals:

• move calculations as deep into hardware as possible for performance and power efficiency
• moldable - have hardware adapt to required computations

Use fpga to encode model weights directly into hardware, allowing for efficient computation and storage. Limited by fpga density and llm model size. With onboard memory caching, the fpga could reconfigure itself for each stage of model evaluation.

FPGAs are computationally efficient, but not as space efficient as harder hardware. Adopting a blade server type form factor would increase compute density.

Use pcie lanes to connect multiple fpga cards to each other, and spread model layers across cards. This would allow for a distributed system where each card handles a different stage of model evaluation for scalability. Management could be performed by separate attached compute ( a la raspberry pi) or on the FPGAs themselves (FPGA-netes, F9s? ).

Good summary of events with links: https://anderegg.ca/2024/10/13/acf-has-been-hijacked

• Guideline invoked: https://github.com/wordpress/wporg-plugin-guidelines/blob/trunk/guideline-18.md
• https://x.com/wp_acf/status/1841843084700598355

Downloaded version 6.3.6 of advanced custom fields from github: [AdvancedCustomFields/acf](https://github.com/AdvancedCustomFields/acf/releases/tag/6.3.6) Downloaded version 6.3.6.2 of secure custom fields from wordpress: https://wordpress.org/plugins/advanced-custom-fields/

> find . -name '*.zip'  -exec sha256sum {} \;
1403316f569fdb8947dbb0ef6f6f0f154678fe978e46ccc06571d71918c685dd  ./advanced-custom-fields.6.3.6.2.zip
91cd88e29e1c220603e17fdfc267943a16e297a87d24325eafc15794364a0a83  ./advanced-custom-fields-6.3.6.zip

Complete txt diff File list txt diff

diff -qr ./advanced-custom-fields-6.3.6/ ./advanced-custom-fields-6.3.6.2/
Files ./advanced-custom-fields-6.3.6/acf.php and ./advanced-custom-fields-6.3.6.2/acf.php differ
Files ./advanced-custom-fields-6.3.6/includes/acf-bidirectional-functions.php and ./advanced-custom-fields-6.3.6.2/includes/acf-bidirectional-functions.php differ
Files ./advanced-custom-fields-6.3.6/includes/acf-helper-functions.php and ./advanced-custom-fields-6.3.6.2/includes/acf-helper-functions.php differ
Files ./advanced-custom-fields-6.3.6/includes/acf-value-functions.php and ./advanced-custom-fields-6.3.6.2/includes/acf-value-functions.php differ
Files ./advanced-custom-fields-6.3.6/includes/admin/admin-internal-post-type-list.php and ./advanced-custom-fields-6.3.6.2/includes/admin/admin-internal-post-type-list.php differ
Files ./advanced-custom-fields-6.3.6/includes/admin/admin.php and ./advanced-custom-fields-6.3.6.2/includes/admin/admin.php differ
Files ./advanced-custom-fields-6.3.6/includes/admin/post-types/admin-field-group.php and ./advanced-custom-fields-6.3.6.2/includes/admin/post-types/admin-field-group.php differ
Files ./advanced-custom-fields-6.3.6/includes/admin/views/acf-field-group/field.php and ./advanced-custom-fields-6.3.6.2/includes/admin/views/acf-field-group/field.php differ
Files ./advanced-custom-fields-6.3.6/includes/admin/views/global/header.php and ./advanced-custom-fields-6.3.6.2/includes/admin/views/global/header.php differ
Files ./advanced-custom-fields-6.3.6/includes/admin/views/global/navigation.php and ./advanced-custom-fields-6.3.6.2/includes/admin/views/global/navigation.php differ
Files ./advanced-custom-fields-6.3.6/includes/admin/views/tools/tools.php and ./advanced-custom-fields-6.3.6.2/includes/admin/views/tools/tools.php differ
Files ./advanced-custom-fields-6.3.6/includes/assets.php and ./advanced-custom-fields-6.3.6.2/includes/assets.php differ
Files ./advanced-custom-fields-6.3.6/includes/class-acf-site-health.php and ./advanced-custom-fields-6.3.6.2/includes/class-acf-site-health.php differ
Files ./advanced-custom-fields-6.3.6/includes/fields.php and ./advanced-custom-fields-6.3.6.2/includes/fields.php differ
Files ./advanced-custom-fields-6.3.6/includes/post-types/class-acf-post-type.php and ./advanced-custom-fields-6.3.6.2/includes/post-types/class-acf-post-type.php differ
Files ./advanced-custom-fields-6.3.6/includes/post-types/class-acf-taxonomy.php and ./advanced-custom-fields-6.3.6.2/includes/post-types/class-acf-taxonomy.php differ
Files ./advanced-custom-fields-6.3.6/README.md and ./advanced-custom-fields-6.3.6.2/README.md differ
Files ./advanced-custom-fields-6.3.6/readme.txt and ./advanced-custom-fields-6.3.6.2/readme.txt differ

diff --color -u --suppress-common-lines -b -r ./advanced-custom-fields-6.3.6/acf.php ./advanced-custom-fields-6.3.6.2/acf.php
--- ./advanced-custom-fields-6.3.6/acf.php	2024-08-28 07:35:13.000000000 -0700
+++ ./advanced-custom-fields-6.3.6.2/acf.php	2024-10-12 17:25:59.560336291 -0700
@@ -1,17 +1,13 @@
 <?php
 /**
- * Advanced Custom Fields
+ * Secure Custom Fields
  *
- * @package       ACF
- * @author        WP Engine
- *
- * @wordpress-plugin
- * Plugin Name:       Advanced Custom Fields
- * Plugin URI:        https://www.advancedcustomfields.com
- * Description:       Customize WordPress with powerful, professional and intuitive fields.
- * Version:           6.3.6
- * Author:            WP Engine
- * Author URI:        https://wpengine.com/?utm_source=wordpress.org&utm_medium=referral&utm_campaign=plugin_directory&utm_content=advanced_custom_fields
+ * Plugin Name:       Secure Custom Fields
+ * Plugin URI:        http://wordpress.org/plugins/advanced-custom-fields/
+ * Description:       Secure Custom Fields is a fork of the Advanced Custom Fields plugin, which will be maintained by WordPress.org, for security and functionality updates.
+ * Version:           6.3.6.2
+ * Author:            WordPress.org
+ * Author URI:        https://wordpress.org/
  * Text Domain:       acf
  * Domain Path:       /lang
  * Requires PHP:      7.4

Notes about changes

• Renames plugin fields but not the zip file
• Still points to https://wordpress.org/plugins/advanced-custom-fields/ , https://wordpress.org/plugins/secure-custom-fields/ doesn’t exist
• pointed support to https://wordpress.org/support/plugin/advanced-custom-fields/
• disables+removes acf pro functions
• removes references to documentation, and help tab
• added code from 6.3.8: Ensure the metabox being called does not perform any unsafe operations
• adjust readme:
  • ACF -> SCF

-ACF helps customize WordPress with powerful, professional and intuitive fields. Proudly powering over 2 million sites, WordPress developers love ACF.
+Secure Custom Fields is a free fork of the Advanced Custom Fields plugin created originally for security updates, but now includes functionality improvements to make this plugin non-commercial in the plugin directory. If you'd like to get involved, submit some code! We want the 2M+ sites that will receive this update to have the best code and functionality possible.

+= 6.3.6.2 =
+*Release Date 12th October 2024*
+* Security - Harden fix in 6.3.6.1 to cover $_REQUEST as well.
+* Fork - Change name of plugin to Secure Custom Fields.
 
-[View the full changelog](https://www.advancedcustomfields.com/changelog/)
+= 6.3.6.1 =
+*Release Date 7th October 2024*
 
-== Upgrade Notice ==
\ No newline at end of file
+* Security - ACF defined Post Type and Taxonomy metabox callbacks no longer have access to $_POST data. (Thanks to the Automattic Security Team for the disclosure)

<code>
== Changelog ==

= 6.3.8 =
*Release Date 7th October 2024*

* Security - ACF defined Post Type and Taxonomy metabox callbacks no longer have access to $_POST data. (Thanks to the Automattic Security Team for the disclosure)

= 6.3.7 =
*Release Date 2nd October 2024*

* Security - ACF Free now uses its own update mechanism from WP Engine servers

= 6.3.6 =
*Release Date 28th August 2024*

* Security - Newly added fields now have to be explicitly set to allow access in the content editor (when using the ACF shortcode or Block Bindings) to increase the security around field permissions. [See the release notes for more details](https://www.advancedcustomfields.com/blog/acf-6-3-6/#field-value-access-editor)
* Security Fix - Field labels are now correctly escaped when rendered in the Field Group editor, to prevent a potential XSS issue. Thanks to Ryo Sotoyama of Mitsui Bussan Secure Directions, Inc. for the responsible disclosure
* Fix - Validation and Block AJAX requests nonces will no longer be overridden by third party plugins
* Fix - Detection of third party select2 libraries will now default to v4 rather than v3
* Fix - Block previews will now display an error if the render template PHP file is not found

The security fix in question is related to how advanced-custom-fields processes GET and POST requests. Files:

• includes/post-types/class-acf-post-type.php
• /includes/post-types/class-acf-taxonomy.php

The first fix, clearing POST parameters, was added as part of acf 6.3.8

Wordpress’s fork took that fix and added clearing the GET parameters as part of it’s first release 6.3.6.2. This didn’t incorporate the switch to pointing to

. despite the changelog referenceing POST data it looks like the change was securing GET requests. Since it uses code from 6.3.8, it must have been available before the “fork” took place. Instead sort of minor update? 6.3.6 -> 6.3.6.2.

> \diff -u --suppress-common-lines -b ./class-acf-post-type-6.3.8.php ./advanced-custom-fields-6.3.6.2/includes/post-types/class-acf-post-type.php
--- ./class-acf-post-type-6.3.8.php     2024-10-12 19:26:56.058355194 -0700
+++ ./advanced-custom-fields-6.3.6.2/includes/post-types/class-acf-post-type.php        2024-10-12 17:25:59.704335853 -0700
@@ -649,12 +649,15 @@
                        }
 
                        $original_post = $_POST; //phpcs:ignore -- Only used as temporary storage to prevent CSRFs in callbacks.
+                       $original_request = $_REQUEST;
                        $_POST         = array();
+                       $_REQUEST         = array();
                        $return        = false;
                        if ( is_callable( $original_cb ) ) {
                                $return = call_user_func( $original_cb, $post );
                        }
                        $_POST = $original_post;
+                       $_REQUEST = $original_request;
                        return $return;
                }

The fix added is also clearing the GET parameters from the original request before calling the meta box callback function specified.

Before calling the function, it grabs it from

$original_cb  = isset( $acf_taxonomy['meta_box_cb'] ) ? $acf_taxonomy['meta_box_cb'] : false;

// Prevent access to any wp_ prefixed functions in a callback.
if ( apply_filters( 'acf/taxonomy/prevent_access_to_wp_functions_in_meta_box_cb', true ) && substr( strtolower( $original_cb ), 0, 3 ) === 'wp_' ) {
  // Don't execute register meta box callbacks if an internal wp function by default.
  return;
}

Stops attacker from being able to pass extra GET parameters to meta box callback functions when setting it for a post. To be able to call this a user must be able to register acf-taxonomy custom post types within WordPress.

// WordPress defaults to the tags/categories metabox, but a custom callback or `false` is also supported.
			$meta_box = isset( $post['meta_box'] ) ? (string) $post['meta_box'] : 'default';

Internally, acf is still used in most places.