Custom POSIX-style Shell (Python)

A shell interpreter built from scratch in Python — no shell libraries, no subprocess module abstraction. Every process spawn, pipe, redirection, and readline hook is wired directly against the os module, following the CodeCrafters "Build Your Own Shell" challenge.

The goal was to actually understand what a shell does at the OS level: process creation, file descriptor plumbing, and terminal I/O.

What it does

• REPL with builtins (cd, pwd, echo, type, exit, history) and external program execution via fork + execvp
• Quote-aware parsing — single quotes, double quotes, and backslash escaping, each with different literal-vs-special-character rules, hand-written as a single-pass character scanner
• I/O redirection — >, >>, 1>, 1>>, 2>, 2>>, implemented differently for builtins (swap sys.stdout) vs. external commands (raw os.open + os.dup2 at the fd level), since only one of those is even running inside the Python process
• N-stage pipelines (cmd1 | cmd2 | cmd3 | ...) — general pipe-chaining using os.pipe() + os.dup2(), with careful fd closing to avoid the classic "reader blocks forever" deadlock
• Tab completion — a stateful callback matching GNU Readline's own completion protocol, covering builtins, $PATH executables, and multi-match / longest-common-prefix behavior
• Command history — history builtin with count-limited output (history -n), plus file persistence (-r/-w/-a, $HISTFILE load-on-start / save-on-exit)

Architecture & design decisions

• OOP refactor early on. Started as a flat script with lambda-based dispatch; refactored into a Shell class once the command-handling logic outgrew a simple if/elif chain. Builtins are stored as a dict[str, Callable], making dispatch a single lookup instead of a growing conditional.
• Redirection as a data-driven map, not branching logic. REDIRECT_OPS maps each operator (>, 2>>, etc.) to a (fd_number, append_bool) tuple, so adding a new redirection operator is a one-line dictionary entry rather than a new code path. This came out of a deliberate refactor once the naive if/elif version got unreadable — see "Notable decisions" below.
• Builtin vs. external redirection are genuinely different mechanisms, not just different code paths for the same idea:
  • Builtins run as plain Python function calls in-process, so redirecting them means temporarily reassigning the Python-level sys.stdout/sys.stderr object.
  • External commands run as separate OS processes, so redirection has to happen at the fd level, before execvp replaces the process image — os.open() the target file, os.dup2() it onto fd 1/2, then exec, because the fd table (unlike everything else) survives both fork() and exec().
• Pipelines use a sliding-window fd handoff. Each iteration creates one new pipe, wires the previous iteration's read-end as this command's stdin, and (for all but the last command) wires this iteration's write-end as its stdout — generalizing cleanly to any pipeline length, not just two commands.
• Completion follows Readline's actual callback contract: complete_command(text, state) is called repeatedly with increasing state until it returns None, mirroring how Readline (and tools like it) lazily generate completion candidates instead of computing the full list upfront.

Design notes

A few specific moments that are more interesting than the feature checklist:

1. The redirection refactor. The first working version of redirection was a wall of near-duplicate if op == '>': ... elif op == '>>': ... elif op == '2>': blocks. Recognizing the duplication and collapsing it into the REDIRECT_OPS mapping + a single dispatch path was a deliberate mid-project refactor.

2. Pipeline fd deadlocks. Every os.pipe() call creates two fds that get duplicated further by fork(). If any stray copy of a pipe's write-end is left open anywhere (parent process included), the reader on the other end blocks forever waiting for input that will never come, because the kernel still thinks a writer might exist. Getting the close-discipline right — parent closes its copy of each pipe end the instant it's handed off to a child.

3. A pipeline stage that's an external command is actually two processes, not one. The pipeline's per-stage child does the fd wiring (dup2/close), then calls the shell's normal execute_command, which — for external programs — forks again internally before execvp. So a 3-command pipeline spins up 6 processes total, not 3. It still works correctly (fd tables survive both fork and exec).

4. Two unrelated history systems coexist. import readline gives up/down-arrow history navigation "for free," entirely inside GNU Readline's own C-level internal list — no code was written for it. Separately, the shell maintains its own self.history Python list, used by the history builtin and file persistence. These two lists are not synchronized and can drift apart.

Setup

./your_program.sh

Requires Python 3.13. See app/main.py for the implementation.

Progress & what's next

Currently 43/76 stages (Base REPL, Navigation, Quoting, Redirection, Command Completion, Pipelines, History, History Persistence). Actively continuing into Filename Completion, Background Jobs, Programmable Completion, and Parameter Expansion.

---

Originally built in 2025; revised and extended in 2026.