BARKS IN THE WIND

 
Rachael: Do you like our owl?
Deckard: It's artificial?
Rachael: Of course it is.
Deckard: Must be expensive.
Rachael: Very. I'm Rachael.

last in the series of guru narrations by kalman reti on genera use. he rehashes a lot of subjects he previously covered, but the core of video is loading an image and then manipulating it through indirect arrays (more...)

this is another guru narration by kalman repi of lisp machine use, this time the goal is produce a cute hack to autoscroll a sheet of music (more...)

Kalman Reti, the last symbolics engineer, did a series of video presentations on using genera. I had those videos laying on my desktop to watch carefully, and extract whatever useful knowledge. There's no system to the notes, it's mostly shortcuts, or snippets of code, of interest only to me. Some things I know and use already, but some were useful first exposure. (more...)

Random usim debug videos I had on my desktop. They are not of interest to anyone, but they document the debugging and fixing of the mouse issues on System 99, that we worked on with ams. (more...)

Some time ago I was experimenting with iPhone's LIDAR/TrueDepth technology, which got me this figurine scan. It's been sitting on my desktop, I figured why not try and figure out how to post it to blog. Here it is in all it's janky glory, (more...)

mtime_of_manifest_file is used to set the mtime from a handful of methods in fossil that are timestamp aware. the function looks up the file in the mlink table which i believe provides the link between EVENT table (i.e. repo history) and VFILE.

src/db.c:896:  int rc = mtime_of_manifest_file(sqlite3_value_int(argv[0]),

db_checkin_mtime_function which is linked to a sql helper function checkin_mtime. The function is documented in a handful of places, including help for sql command. checkin_mtime(X,Y) Return the mtime for the file Y (a BLOB.RID) found in check-in X (another BLOB.RID value).

src/file.c:2095:** mtime_of_manifest_file() is asked to provide the timestamp for a
src/file.c:2319:      if( newMtime || mtime_of_manifest_file(vid, fid, &newMtime)==0 ){
src/file.c:2364:        if(mtime_of_manifest_file( vid, fid, &newMtime )!=0){

The context of above is touch_cmd which knows how to set mtime using following options,

  --now          Stamp each affected file with the current time.
                 This is the default behavior.
  -c|--checkin   Stamp each affected file with the time of the
                 most recent check-in which modified that file.
  -C|--checkout  Stamp each affected file with the time of the
                 currently-checked-out version.

src/checkin.c:186:   * SQLite opcode.  checkin_mtime() calls mtime_of_manifest_file() which
src/vfile.c:249:      if( mtime_of_manifest_file(vid,rid,&desiredMtime)==0 ){

Above is part of vfile_check_signature, which the main function that figures out if the file has been changed in the working tree. It syncs the working tree against fossil's VFILE table. The table also keeps mtime's for the purposes of quickly figuring out what has changed.

The users of vfile_check_signature make particularly strong assumptions about the tie between checkin time and modification time e.g. fossil ls --age is described as Show when each file was committed. and is essentially equivalent to the following fossil sql,

select
pathname,
datetime(checkin_mtime((select value from vvar where name='checkout'), rid), 'unixepoch', toLocal())
from vfile

but of course before doing that sql fossil does vfile_check_signature to ensure that sql state reflects filesystem state.

create_manifest used from commit_cmd and manifest_parse

ref fossil hacker howto

A month ago my key has expired again, so I spent some time exploring subkey behavior in gnupg. This is an experimental procedure for converting a typical primary/sub key pair into a pair of independent keys. I don't have practical experience with all the different implications of the switch, so proceed at your own risk.

First and most important point is that contrary to my previous assumption pgp can work with a single, primary key.

Default key generator though forces you into a particular set of ancient assumptions, that also gave us keychain databases, trust levels, signing parties and other such rituals, that seem to have been designed in vacuum, and failed to gain standing in the real world. A key can be used for Signing and Encryption, but also Certification, that is the signing of subkeys in a keychain. What the key is allowed to do is stored in the key's usage flags, and is enforced by gnupg. The usage is split between primary key and any number of subkeys. By default the primary key is allowed to Sign and Certify, the subkey in turn can only Encrypt. Gnupg gives preference to subkeys, so if e.g. a subkey has a Sign flag, it will be used for signing instead of the primary key.

Here's an example of a typical primary/sub pair, that you get with gen-key:

% gpg --homedir keys --edit-key 60D55B29
Secret key is available.

pub  2048R/60D55B29  created: 2018-10-12  expires: never       usage: SC
                     trust: ultimate      validity: ultimate
sub  2048R/4FC43EBB  created: 2018-10-12  expires: never       usage: E
[ultimate] (1). test key

The goal is to remove the subkey, and allow the primary key to also encrypt. You can delete the subkey using edit-key interface, but the procedure for editing usage flags is more elaborate. Unfortunately I haven't found a way to edit key's usage flags after the key has been generated, but there's a way to surgically split the keys and give them whatever options you need. Frankly I'm surprised there isn't an easier way!¹

A pgp key consists of a series of packets,

% gpg --homedir keys --export-secret-key | pgpdump |grep Packet
Old: Secret Key Packet(tag 5)(920 bytes)
Old: User ID Packet(tag 13)(16 bytes)
Old: Signature Packet(tag 2)(312 bytes)
Old: Secret Subkey Packet(tag 7)(920 bytes)
Old: Signature Packet(tag 2)(287 bytes)

Secret key/subkey packets contain the necessary values to do RSA computation and nothing else. The only difference between a key and a subkey is the packet tag.

Old: Secret Key Packet(tag 5)(920 bytes)
        Ver 4 - new
        Public key creation time - Fri Oct 12 00:29:24 EDT 2018
        Pub alg - RSA Encrypt or Sign(pub 1)
        RSA n(2048 bits) - ...
        RSA e(17 bits) - ...
        RSA d(2044 bits) - ...
        RSA p(1024 bits) - ...
        RSA q(1024 bits) - ...
        RSA u(1023 bits) - ...
        Checksum - 3f b4

A user id packet is a string value that's assembled out of the relevant gen-key questions,

Old: User ID Packet(tag 13)(16 bytes)
        User ID - test key

A signature packet is the bucket full of Everything Else,

Old: Signature Packet(tag 2)(312 bytes)
        Ver 4 - new
        Sig type - Positive certification of a User ID and Public Key packet(0x13).
        Pub alg - RSA Encrypt or Sign(pub 1)
        Hash alg - SHA1(hash 2)
        Hashed Sub: signature creation time(sub 2)(4 bytes)
                Time - Fri Oct 12 00:29:24 EDT 2018
        Hashed Sub: key flags(sub 27)(1 bytes)
                Flag - This key may be used to certify other keys
                Flag - This key may be used to sign data
        Hashed Sub: preferred symmetric algorithms(sub 11)(5 bytes)
                Sym alg - AES with 256-bit key(sym 9)
                Sym alg - AES with 192-bit key(sym 8)
                Sym alg - AES with 128-bit key(sym 7)
                Sym alg - CAST5(sym 3)
                Sym alg - Triple-DES(sym 2)
        Hashed Sub: preferred hash algorithms(sub 21)(5 bytes)
                Hash alg - SHA256(hash 8)
                Hash alg - SHA1(hash 2)
                Hash alg - SHA384(hash 9)
                Hash alg - SHA512(hash 10)
                Hash alg - SHA224(hash 11)
        Hashed Sub: preferred compression algorithms(sub 22)(3 bytes)
                Comp alg - ZLIB (comp 2)
                Comp alg - BZip2(comp 3)
                Comp alg - ZIP (comp 1)
        Hashed Sub: features(sub 30)(1 bytes)
                Flag - Modification detection (packets 18 and 19)
        Hashed Sub: key server preferences(sub 23)(1 bytes)
                Flag - No-modify
        Sub: issuer key ID(sub 16)(8 bytes)
                Key ID - 0xC1E46ED560D55B29
        Hash left 2 bytes - 1f 27
        RSA m^d mod n(2048 bits) - ...
                -> PKCS-1

It contains and signs various dates, flags, and preferences. Of interest are the "key flags". Now one could write a pgp packet editor, that will patch the relevant bits in, and I suspect that's something that asciilifeform might have on his workbench already, but there exists a somewhat odd way of breaking the keys apart and then reassembling them from bits.

We will need a signature with the right key flags enabled. I said that gnupg gives you a primary/sub key pair by default, but with an --expert flag you can make it produce whatever key configuration you want, and that should really be the recommended way of making new keys in the republic.

% mkdir tmp
% gpg --homedir tmp --gen-key --expert
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
Your selection? 8

Possible actions for a RSA key: Sign Certify Encrypt Authenticate
Current allowed actions: Sign Certify Encrypt

   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? q
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 1024
[...]

Now we split the two key sets we have with a gpgsplit utility that's part of gnupg,

% gpg --homedir keys --export-secret-key | gpgsplit -p k
% gpg --homedir tmp --export-secret-key | gpgsplit -p tmp
% ls
k000001-005.secret_key    k000005-002.sig           tmp000002-013.user_id
k000002-013.user_id       keys                      tmp000003-002.sig
k000003-002.sig           tmp
k000004-007.secret_subkey tmp000001-005.secret_key

which gives us all the packets in separate files. The ones prefixed with k- are the original keys (there's 5 files that correspond to 5 packets from pgpdump above), the tmp- ones are the temp key we just generated.

An arbitrary key combination can now be assembled out of the individual packets, but we use original primary key, original user id and the signature from the temp key.

% cat k000001-005.secret_key k000002-013.user_id tmp000003-002.sig > key

A straight import of the resulting key will fail, because of the invalid signature, but passing an --allow-non-selfsigned-uid will bypass the signature verification, while still applying whatever preferences are stored in signature packet,

% gpg --homedir tmp1 --allow-non-selfsigned-uid  --import key
gpg: WARNING: unsafe permissions on homedir `tmp1'
gpg: keyring `tmp1/secring.gpg' created
gpg: keyring `tmp1/pubring.gpg' created
gpg: key 60D55B29: secret key imported
gpg: key 60D55B29: accepted non self-signed user ID "test key"
gpg: tmp1/trustdb.gpg: trustdb created
gpg: key 60D55B29: public key "test key" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
gpg: no ultimately trusted keys found

Finally in order to fix the key's internal consistency we need to delete the bogus signature and re-sign the identity,

% gpg --homedir tmp1 --allow-non-selfsigned-uid --edit-key 60D55B29
Secret key is available.

pub  2048R/60D55B29  created: 2018-10-12  expires: never       usage: SCEA
                     trust: unknown       validity: unknown
[ unknown] (1). test key

gpg> uid 1

pub  2048R/60D55B29  created: 2018-10-12  expires: never       usage: SCEA
                     trust: unknown       validity: unknown
[ unknown] (1)* test key

gpg> delsig
uid  test key
sig?3        4FEE77E7 2018-11-13
Delete this unknown signature? (y/N/q)y
Deleted 1 signature.

gpg> sign

pub  2048R/60D55B29  created: 2018-10-12  expires: never       usage: SCEA
                     trust: unknown       validity: unknown
 Primary key fingerprint: 454C 1EFC A29D 02A0 E6CE  3A47 C1E4 6ED5 60D5 5B29

     royal astronomer

Are you sure that you want to sign this key with your
key "test key" (60D55B29)

This will be a self-signature.

Really sign? (y/N) y

gpg> save

The entire ungodly procedure gives us the original primary key 60D55B29 with all the usage flags enabled. The procedure can be repeated with the subkey, and is left as an exercise for the reader. It requires first patching k000004-007.secret_subkey subkey's first byte to 149 to switch it from Secret Subkey Packet(tag 7) to Secret Key Packet(tag 5).²

Now one can distribute the primary key as the canonical key, and keep the newly elevated subkey around to decrypt messages from the correspondents who are using an old pubkey.

1. I believe gpg2 allows you to edit usage flags, but since I don't have it anywhere on my machines I can't confirm it. [↩]
2. I've used the following, trivial Ada tool,

   with Ada.Direct_IO;
   with Ada.Command_Line; use Ada.Command_Line;
   procedure P is
      type Byte     is mod 2**8;
      package Io is new Ada.Direct_IO(Byte); use Io;
      F: Io.File_Type;
      B: Byte;
      Position: Positive_Count;
      Wrong_File: exception;
   begin
      Open( F, Inout_File, Argument(1) );
      Position := Positive_Count'Value( Argument(2) );
      B := Byte'Value( Argument(3) );
      Write( F, B, Position );
      Close( F );
   end;
   

   which can be used thus,

   % gprbuild p.adb
   
   [...]
   
   % ./p 000004-007.secret_subkey 1 149
   
   % pgpdump 000004-007.secret_subkey | grep Packet
   
   Old: Secret Key Packet(tag 5)(920 bytes [↩]

As has been discussed in the logs, vtools doesn't stand on its own as a V implementation. Instead it's a collection of tools for working with vpatches. V authors can use vtools so as to not rely on often brittle GNU utilities.

On my own workbench I've been using a patched up version of original asciilifeform's v.py, where I replaced a call to GNU patch with one to vpatch. The replacement is essentially a drop in¹, with the advantage of being much stricter about the vpatches that are accepted and also making sure that the press hashes are valid.

I have barely touched v.py otherwise², so I consider this a proof of concept release. It consists of two patches, the original v.py version 99 genesis³ and my own modifications.

Unless you have a working keccak v, you'll need to bootstrap manually. Assuming you have a working vtools build,

PATH=path to vtools:$PATH
mkdir {wot,seals,patches}
curl --silent http://wot.deedbot.org/BDDE12104FE81BE7F83B698F5356DE4752432A9E.asc -o wot/phf.asc
gpg --import wot/phf.asc
curl --silent -o patches/v99.vpatch  http://btcbase.org/data/vpy/v99.vpatch
curl --silent -o patches/v98.vpatch  http://btcbase.org/data/vpy/v98.vpatch
curl --silent -o seals/v99.vpatch.phf.sig  http://btcbase.org/data/vpy/v99.vpatch.phf.sig
curl --silent -o seals/v98.vpatch.phf.sig  http://btcbase.org/data/vpy/v98.vpatch.phf.sig
gpg --verify seals/v99.vpatch.phf.sig patches/v99.vpatch
gpg --verify seals/v98.vpatch.phf.sig patches/v98.vpatch
cat patches/v99.vpatch patches/v98.vpatch | vpatch
pip install python-gnupg
chmod +x v/v.py
./v/v.py --wot ./wot -fingers --seals ./seals ./patches p ./patches/v98.vpatch v_press

You now have a self-pressed v.py in the v_press directory!

Some things to note: the bulk of bootstrapping effort is verifying the patch signatures, something that v does for you. On the other hand you can just cat any number of patches into vpatch utility and it will produce a verified press. Asciilifeform's v.py uses stock python, but it does depend on python-gnupg package, which can be installed through pip or whatever global packaging system (on gentoo it's emerge python-gnupg).

1. right now vpatch doesn't support target directory, and presses into the current directory, so I had to do some changes to accommodate v.py's concept of destination. i also hardened the call out to an external process, though perhaps unnecessarily. [↩]
2. I have also added support for subkeys. [↩]
3. the first release of v.py is actually version 100, but the diff between 100 and 99 is in my opinion entirely cosmetic, so I avoided pedantically reconstructing the entire chain, and started with a canonical version of v.py. [↩]

I'm going to call this post a vtools pre-release. I'm deferring the proper release write up till Wednesday, but meanwhile the relevant release work has been done, and it's good time to point interested parties to the bits so that further log discussion can happen. I doubt that my write ups stand on their own, that is without also close following of the going ons in the logs, but this post is particularly so only of interest to specific people.

I've reground the project around manifest file. From previous conversations, it seems like the format is mostly inconsequential, so I'm using <date> <nick>\n<message>. An example of the manifest file press, you can see the implicit press order in btcbase annotation, and how that manifest change looks in a vpatch file. In the process I've discovered that btcbase presser isn't working quite right, so at the moment /tree/ shouldn't be relied on for exploration of the press.¹

Keccak vdiff /vpatch are now at feature parity with the existing shell based tooling, specifically vpatch now supports no newline directive. We're going to start working with a complete round trip in mp-wp, which is going to be keccak only release. I would still like to make vpatch work with SHA-512 though.

Current complete patchset, with vtools_vpatch_newline keccak and vdiff_sha_static SHA-512 heads,

http://btcbase.org/data/vtools/keccak.vpatch
http://btcbase.org/data/vtools/keccak.vpatch.phf.sig
http://btcbase.org/data/vtools/vdiff_fixes_newline_gcc.vpatch
http://btcbase.org/data/vtools/vdiff_fixes_newline_gcc.vpatch.phf.sig
http://btcbase.org/data/vtools/vdiff_keccak.vpatch
http://btcbase.org/data/vtools/vdiff_keccak.vpatch.phf.sig
http://btcbase.org/data/vtools/vdiff_sha_fixes_newline_gcc.vpatch
http://btcbase.org/data/vtools/vdiff_sha_fixes_newline_gcc.vpatch.phf.sig
http://btcbase.org/data/vtools/vdiff_sha_static.vpatch
http://btcbase.org/data/vtools/vdiff_sha_static.vpatch.phf.sig
http://btcbase.org/data/vtools/vtools_fixes_bitrate_char_array.vpatch
http://btcbase.org/data/vtools/vtools_fixes_bitrate_char_array.vpatch.phf.sig
http://btcbase.org/data/vtools/vtools_fixes_static_tohex.vpatch
http://btcbase.org/data/vtools/vtools_fixes_static_tohex.vpatch.phf.sig
http://btcbase.org/data/vtools/vtools_genesis.vpatch
http://btcbase.org/data/vtools/vtools_genesis.vpatch.phf.sig
http://btcbase.org/data/vtools/vtools_vdiff_sha.vpatch
http://btcbase.org/data/vtools/vtools_vdiff_sha.vpatch.phf.sig
http://btcbase.org/data/vtools/vtools_vpatch.vpatch
http://btcbase.org/data/vtools/vtools_vpatch.vpatch.phf.sig
http://btcbase.org/data/vtools/vtools_vpatch_newline.vpatch
http://btcbase.org/data/vtools/vtools_vpatch_newline.vpatch.phf.sig

1. e.g. vtools_vpatch_newline's manifest contains extra two entries at the end, which is a result of a buggy press rather than the contents of relevant vpatches [↩]