TL;DR

Start keeping track of your mistakes. “Mistakes”, here refer to what you’ve learnt but not what you’ve abandoned.

  • Either add them to your existing blog or start blogging now.
  • Or just Keep track of them in a dedicated git repository
  • Or Use symlinks or git worktree to connect it to your project.

My suggestion

  • I recommend a docs/learnings directory in each of your projects gitignored and symlinked to the dedicated repo and hosting it as a blog for you to revisit.

Introduction

To err is human.

Mistakes you’ve made are the only valid proof that you’ve learnt something.

The most important part in software development or any kind of learning, is learning from past mistakes.

Github/Gitlab issues, PRs are widely used and serve as good references to keep track of the exact development process, but keeping track of your learnings will be extremely useful on top of that.

All the silly mistakes you’ve made, wrong decsicions, or some very useful/new ways of doing something. They will all be at one place neatly categorized, tagged, sorted so that one day you could see just how much you’ve learned.

It will be exteremly useful for future reference, to understand just what bug you faced and why it crept in in the first place and what your journey i.e. the train of thought and your google searches, were in the process of fixing it.

All of the above can be tracked while including proper references and search terms to get back to the exact development enviroment and mindset most efficiently.

Why

Your visits to stackoverflow can be predictable.

Let’s look at some q&as

  • Learning a new Lang/Framework/Pattern? Very high
  • Something broke? Moderate
  • Found a bug? Moderate
  • How to run a command? Definitely (Yeah I said it, I read documentation through stackoverflow)
    • eg. ffmpeg, git commands (git commands have millions of views per qn on stackoverflow)
  • Copied a link to stackoverflow post in a code comment? Yes
    • Made some changes or extended it? Yes (most likely)
    • Need to revisit it again? Yes
  • Found some question, but no answers? Ocassionally
    • Found a way to solve it from another source? Most likely
    • Went back and answer on stackoverflow? Rarely
  • etc. this is not a post on stackoverflow and its uses, so let’s move on.

My point being, as you see from the above, a developer cannot live without stackoverflow, generally speaking.

So if you need a history of your development i.e. the journey to get the exact mindset back, it’s quite time consuming.

Q: Soultion? A: Very simple, right? Just keep track of it.

What can be tracked

I can only suggest what I personally track

  • Interesting bugs
    • Reason for occurance
    • Final solution
    • Journey : list of urls
  • Obvious/Silly/Stupid mistakes
    • Don’t forget these, track them specially
    • They hurt everyone involved, including you
  • Very useful code snippets
    • eg. I use these repeatedly
      • get os name in bash (see #2 below unameOut switch case)
      • verbose bash commands (see #2 below _exe)
  • Higly visited or bookmarked stackoverflow posts
    • Any stackexchange site or url for that matter

Here is a way to keep track of your “learnings”

A simple way to track

There are 3 steps, each step is independant so you can stop reading right away at any step, if you got the general idea :star-struck: or if this is straight up boring :cry:.

You can use any combinations (intersection) from the 3 steps and it would work.

  1. Write a blog

    • It can be public/private
    • Optionally sync with github/gitlab
    • And can also host somewhere either gh-pages(public) / locally(private)

  2. A git repo dedicated to learnings

    • You need to remember to track your learnings
      • Which for me is not trivial (TODO address this later)
    • No blogs here but just markdown and cross links
    • Can also make it a blog by using something like hugo
    • Transitioning to use hugo should be smooth enough

  3. Keep a directory docs/learnings under every project you work on

    • If it is a personal project or a project where sharing your learnings with colleagues doesn’t do anything bad.
      • Let it remain in the repository and consider this step’s idea conveyed.
    • Otherwise gitignore docs/learnings and track them separately
      • One way is to create symlinks, hard link to the files
      • Alternate way, track in a branch in your fork which along side git worktree will make it as simple as just writing files into a subdirectory

Example/Tutorial

Now we will have them all setup and let’s weigh pros and cons at the end.

Note: Pros and cons section here maybe biased, it’s up to you to decide which one works best for you.

1. Writing a blog

That’s what I’m doing right now. (But I’m doing it via method #2)

Now there are many predecessors who’ve done it and also made it very easy. You can use Medium, Wordpress, Blogger, Tumblr etc.

Developers afaik tend to use Medium or host their own blogs.

I’ve tried Blogger it was extremely simple to setup

Pros

  • Google owned
  • Google UI
  • Google login
  • Google website visitors login

Cons

  • Same as the pros
  • Ancient

Tumblr is a dead man walking, avoid it if possible. Setup takes time.

Medium has some visitor monthly restriction B.S. Never tried to publish anything over Medium.

  • UI very good
  • Highly used
  • Highly used by developers

Wordpress

  • Ancient
  • 75 million websites use it

Publicize my mistakes. Really?

Now should the blog be public? It doesn’t have to be.

Making your blog public comes with added advantages and disadvantages.

Pros:

  • If you get many visitors, better start a yt channel
  • You’ll get free advices, it is up to you to filter them
  • You would definitely help atleast one human being other than yourself.

Cons:

  • If no one cares/comments on your blog you might feel discouraged.
  • If you get negative criticism or hate comments you might get discouraged.
  • If you get inappropriate comments on your blog, you need to moderate them.

Private blog?

If you are in no place to decide your project’s status or there’s some NDAs (non-disclouse agreements) involved, you obviously won’t host it even for public/private/internal consumption. (Nice knowing you!)

Of course you can recreate and modify the project to only include your learnings. You won’t be breaking the NDA but it is risky nevertheless.

You’re not stupid, you know what can be acceptable, if not move on.

Otherwise i.e. if you think you won’t get in any trouble if personal use is allowed, make a private blog. Host it somewhere, password protect it, only you read it.

2. A dedicated git repo

You can track your content in a dedicated git repo with some structure

Say you can do something like

learnings(root)/
    - .git/
    - scripts/
        - sync.sh # or sync.bat which deploys, updates the symlinks etc.
    - current/ -> 2021/05/ (symlink) # need to update regularly
    - 2021/
        - 02/
        - 05/
            - nice-3-books.md
            - images/
    - 2020/
    - # ...

Here you need to worry about creating a new yyyy/mm every time you write a new post.

Let’s use a symlink to fix that. The current/ will always point to the current date’s post so you can just add in any files in that folder itself.

But symlinks won’t change unless we change them so we will write a script.

Why

Q: Now why would you do something like syncing to a github/gitlab repo? A: Learn about git and version control

Q: Is this better than a blog? A: Yes, if it is private and maybe not so for public

Example

Note: This section can be skipped entirely but is a good example nonetheless. It turned out very long so I wouldn’t wanna bore you. Read it at the end if you wish.

Now consider the above layout.

Say we have write our scripts like this scripts/sync.sh for Unix machines

# file: sync.sh
_exe(){
    # https://stackoverflow.com/a/64644990/8608146
    [ $1 == on  ] && { set -x; return; } 2>/dev/null
    [ $1 == off ] && { set +x; return; } 2>/dev/null
    echo + "$@"
    "$@"
}
exe(){
    { _exe "$@"; } 2>/dev/null
}

exe on # fancy set -x

# commit changes (change or remove this part if not needed)

git pull
git status
git add .
git commit -m "$(date) auto update"
git status
git push

exe off

# below part updates the ./current/ to point ./yyyy/mm

unameOut="$(uname -s)"
case "${unameOut}" in
    Linux*)     machine=Linux;;
    Darwin*)    machine=Mac;;
    CYGWIN|MINGW|MSYS*)
        machine=Windows
        EXT=".exe"
    ;;
    *)          machine="UNKNOWN:${unameOut}"
esac

if [ $machine == "Windows" ]; then
    # Windows
    echo "Use the batch script on Windows"
else
    # Linux, Darwin(mac), etc.
    date=$(date '+%Y/%m')
    exe on
    mkdir -p $PWD/$date current

    ln -s $PWD/$date current/
    exe off
fi

The above code is straight forward except that exe thing which can be removed completely and the script’s functionality will not change.

Now for Windows machines scripts\sync.bat

@rem file: sync.bat

@rem https://stackoverflow.com/a/1445724/8608146
set dtStamp=%date:~-4%\%date:~4,2%
set destdir=%cd%\%dtStamp%
set backupd=%date:~-4%_%date:~4,2%_%date:~7,2%_0%time:~1,1%_%time:~3,2%_%time:~6,2%

@rem commit changes (change or remove this part if not needed)

git pull
git status
git add .
git commit -m "%backupd% auto update"
git status
git push

@echo off

if not exist "%destdir%" mkdir "%destdir%"
if not exist "backups" mkdir "backups"
if not exist "current" mkdir "current"

@rem for for soft links
set softlinks=y
@rem for hard links
@rem set softlinks=

@rem If using softlinks leave this commented
if defined softlinks (
    echo "Using soft links, you may lose data if you accidentally delete files"
    if exist "current" rmdir current
) else (
    @rem if using hard links MUST leave it commented out
    @rem uncommenting will move it to backups
    @rem instead of deleting, just incase

    @rem if exist "current" mv current backups\current_%backupd%
    @rem create empty current again
    echo "Using hard links, you have a backup in current/ and also yyyy/mm"
    if not exist "current" mkdir "current"
)

@rem Make softlink this works, no issues
@rem except if you delete something in current you'll lose it forever
@rem eg. if not tracking or working changes or rm command
if defined softlinks (
    mklink /D current %destdir%
) else (
    @rem Make hardlink (not possible for directories /J is not working)
    @rem mklink /J current %destdir% @rem this does not work
    @rem as of now just hardlinking recursively and forfiles with cmd is super slow
    set basedir=%cd%
    cd %destdir%

    @rem https://stackoverflow.com/a/12209020/8608146
    @rem loop over all new directories and create them
    for /f "tokens=1* delims=\" %%A in (
    'forfiles /s /m * /c "cmd /c if @isdir==TRUE echo @relpath"'
    ) do (
        for %%F in (^"%%B) do (
            echo %%F
            if not exist "%basedir%\current\%%~F" mkdir "%basedir%\current\%%~F"
        )
    )

    @rem loop over all files and hard link them individually
    for /f "tokens=1* delims=\" %%A in (
    'forfiles /s /m * /c "cmd /c if @isdir==FALSE echo @relpath"'
    ) do (
        for %%F in (^"%%B) do (
            if not exist "%basedir%\current\%%~F" mklink /H "%basedir%\current\%%~F" %%~fF
        )
    )

    cd ..\..
)

Now that’s a lot of code for symlinking. Unfortunately on Windows, directory hard links are not supported, so I had to write recursive hard links for each file separately (in batch :cry: It took too long!).

So you open your favorite ide/editor and open current, add files/folders/posts etc. and run bash ./scripts/sync.sh or .\scripts\sync.bat

On windows prefer soft links, uncomment set softlinks=y

On linux this script is untested as of now TODO test it.

Caveats with this approach:

  • If you renamed a file it’s assumed to be deleted and created again
    • So renaming will create duplicates you need to remove the older one carefully
  • Need to run scripts every time

Note: If you find this hard links hard to manage, assume they don’t exist and proceed with soft linking which is referred to as normal symlinks.

Q: What are these symlinks? A: Symlinks are path shortcuts.

  • Soft links: A soft link is a pointer to a dir/file which upon deletion does nothing to the original but if it’s a directory its contents when deleted directly delete the orginal
  • Hard links: A hard link is a clone of the original which points to the exact same space in disk as the original. So any changes to it or the original will get synced by the hardware itself. If you delete one other stays, no matter which one.

Q: Why these symlinks? A: To not worry about which date today is and find the exact directory to put your files in.

Can we do better?

Yes! The above mentioned quickly became too complex. Which finally turned out to be directory syncing. Which sounds an awful lot like something we know, yes version control, git!

That’s what we will jump to in the next section.

3. Dedicated learnings subdirectory

The idea is simple, say we have a project that we’d like to track our learnings on. yourproject

yourproject/
    - .git
    - .gitignore
    - src/
    - build/
    # etc ...
    - docs/learnings # create this and gitignore it

Add docs/learnings or whatever to your project’s gitignore.

Then you have two options.

  • Symlink (soft > hard) to some file or folder
  • git worktree into the docs/learnings directory.
    • You don’t need any scripts to manage the things here, unlike above i.e. when using a hard link approach, it works fine with soft links.

Now we need to have one more repo say learnings

learnings/
    - .git
    - projects/
        - yourproject-learnings.md
        - otherproject.md
        # OR some other way of orgnizing
        - yourproject/
            - images/
            - readme.md
            # etc., any kind of organization

Symlinking

If we want to symlink its just as simple as a single command

Unix (Linux, Mac)

# directory
ln -s "/abs/path/to/learnings/projects/yourproject" "../yourproject/docs/learnings"
# or file
ln -s "/abs/path/to/learnings/projects/yourproject-learnings.md" "../yourproject/docs/learnings/readme.md"

On Windows

# directory
mklink /D "D:\abs\path\to\learnings\projects\yourproject" "..\yourproject\docs\learnings"
# file can be hard link or soft. But for a file hard link is better
mklink /H "D:\abs\path\to\learnings\projects\yourproject-learnings.md" "..\yourproject\docs\learnings\readme.md"

Pros

  • Simple symlink commands run just once
  • Git does everything else
    • Just git push both project and the repo

Cons

  • Hard Symlinking with directories not possible on Windows

Git worktree

Now this is interesting

I’ve recently found out about this command when researching for this blog post.

It is an alternate git work flow. Instead of git checkout some-branch you’ll do git worktree add ../path/to/dest some-branch and it will checkout the some-branch in the provided path.

But it comes with a very cruel restriction. You can’t checkout the same branch at different locations.

So for us, you probably saw where this is going.

Yes, a separate branch for each project (:exploding_head:)

cd learnings/
git status # then git stash or something
git branch branch-yp # creates a new branch, no need to checkout

# checkout the branch at the docs/learnings residing in the project
git worktree add ../yourproject/docs/learnings branch-yp # branch name

# list all worktrees
git worktree list

Now just update this repo along with the project.

Optionally merge this branch in to the main branch via PRs. Then you’ll have all of your learnings at one place, on the repo in the main branch.

Pros

  • Setting these up just once for each project at the very beginning.
  • Compared to symlinking before, here there is no such thing
  • Git does the 💯% of the job. (only when you say so of course)
    • Just git push both project and the repo

Cons

  • Need to remember to use git worktree list I guess.

Combining all the above

Using with hugo

Hugo is a static site generator. Very fast to setup and running.

Do the steps 1, 3 and you’re good to go.

Deploy on Netlify or ghpages

Mangaging time

Q: Now how long should I spend for this task? A: On average a blog post takes 3h 57m to make. I agree that it is too much time for something only you would read in the future. So just throwing this out there, maybe halve it i.e. 1h 58m 30s?

Q: Is this worth my time? A: Don’t know, Unfortunately I’m not sure how practical this is. I was only an undergrad when I’ve started this tracking, my point being I had time.

Q: So? A: Try it out?

Alternatives

I have had some experience with Confluence, Jira, Notion, Trello they all offer very good and advanced features. If you like ’em UIs use them. And you are probably already using them at work.

I hope so far it has been useful for terminal/vim dev friendly as well.

Conclusion

Start tracking learnings, this is your personal changelog.

Dedicating your time to this, will definitely pay off in the future.

Next steps

  • Note: this is regarding the future of this post
  • I plan to add a software section which makes all of the above mentioned process easy, like one click deploys.
  • If no such thing exists, after taking a look at your suggestions let’s build it.
    • Will make a part-2 then.