kilko.de - swift

Decoding Unknown Data with Codable

2021-09-01T00:00:00+00:00

A friend recently asked me how to decode the following JSON through Swift's Codable</code> type.</p>

{</span></span>
  "</span>a</span>"</span>: </span>1</span>,</span></span>
  "</span>b</span>"</span>: </span>"</span>foobar</span>"</span>,</span></span>
  "</span>option.extra</span>"</span>:</span> {</span></span>
    "</span>a</span>"</span>:</span> 1</span></span>
  }</span>,</span></span>
  "</span>option.extra.more</span>"</span>: </span>4</span></span>
}</span></span></code></pre>
The problem is that only the first two keys (a</code> and b</code>) are guaranteed to exist, anything aside from that (here everything prefixed with option</code>) is extra and unknown at compile-time. Unknown keys should not be discarded however, they should be decoded into a dictionary for later use, we want a type like this.</p>
struct</span> Json</span> {</span></span>
  var</span> a:</span> Int</span></span>
  var</span> b:</span> String</span></span>
  var</span> extraOptions:</span> [</span>String</span>:</span> ?</span>]</span></span>
}</span></span></code></pre>
It's a bit unclear what type to use for extraOptions</code>, since we don't really know what the values are. They can be anything, as long as it's decodable. The folks at Flight School</a> have a fantastic package for us to facilitate that conveniently called AnyCodable</a>, which provides type-erased wrappers for Encodable</code>, Decodable</code>, and Codable</code> values. We can use that to represent the requirement of having anything</em>, as long as it's decodable, by defining the type as follows.</p>
struct</span> Json</span> {</span></span>
  var</span> a:</span> Int</span></span>
  var</span> b:</span> String</span></span>
  var</span> extraOptions:</span> [</span>String</span>:</span> AnyDecodable</span>]</span></span>
}</span></span></code></pre>
As it stands we're getting the automatically synthesized implementation of Json</code>'s initializer, which will try to find a dictionary called extraOptions</code> in the source JSON, which obviously isn't present. We'll have to implement the initializer ourselves. Doing so will also require us to define a type conforming to CodingKey</code>, which we can use to specify the encoded keys in our source data.</p>
extension</span> Json</span>:</span> Decodable</span> {</span></span>
  private</span> enum</span> CodingKeys</span>:</span> CodingKey</span> {</span></span>
    case</span> a</span>, </span>b</span></span>
  }</span></span>
</span>
  init</span>(</span>from</span> decoder</span>: </span>Decoder</span>)</span> throws</span> {</span></span>
    let</span> container </span>=</span> try</span> decoder.</span>container</span>(</span>keyedBy</span>:</span> CodingKeys.</span>self</span>)</span></span>
    self</span>.</span>a</span> =</span> try</span> container.</span>decode</span>(</span>Int</span>.</span>self</span>, </span>forKey</span>:</span> .</span>a</span>)</span></span>
    self</span>.</span>b</span> =</span> try</span> container.</span>decode</span>(</span>String</span>.</span>self</span>, </span>forKey</span>:</span> .</span>b</span>)</span></span>
    self</span>.</span>extraOptions</span> =</span> [</span>:</span>]</span></span>
  }</span></span>
}</span></span></code></pre>
We've now run into the problem of somehow having to specify all keys in our CodingKeys</code> enum, which obviously doesn't help for unknown keys. What we can do though is to use the CodingKey</code> protocol to create our own dynamic</em> key handling type, which can handle any String</code> as a key.</p>
struct</span> DynamicCodingKeys</span>:</span> CodingKey</span> {</span></span>
  var</span> stringValue:</span> String</span></span>
  init?</span>(</span>stringValue</span>: </span>String</span>)</span> {</span></span>
    self</span>.</span>stringValue</span> =</span> stringValue</span></span>
  }</span></span>
</span>
  //</span> not used here, but a protocol requirement</span></span>
  var</span> intValue:</span> Int</span>?</span></span>
  init?</span>(</span>intValue</span>: </span>Int</span>)</span> {</span></span>
    return</span> nil</span></span>
  }</span></span>
}</span></span></code></pre>
While we could use this entirely instead of our previous implementation, I personally find it quite a bit cleaner to keep the existing CodingKeys</code> for known keys, but cover everything else with the DynamicCodingKeys</code>. Creating a new container from our decoder keyed with DynamicCodingKeys</code> however will give us a container containing all</em> keys, not just the extra ones, so we'll have to filter out those we've already taken care of, leaving us with something like this.</p>
extension</span> Json</span>:</span> Decodable</span> {</span></span>
  private</span> enum</span> KnownCodingKeys</span>:</span> CodingKey</span>, </span>CaseIterable</span> {</span></span>
    case</span> a</span>, </span>b</span></span>
</span>
    static</span> func</span> doesNotContain</span>(</span>_</span> key</span>: </span>Json.DynamicCodingKeys</span>)</span> -></span> Bool</span> {</span></span>
      !</span>Self</span>.</span>allCases</span>.</span>map</span>(</span>\.</span>stringValue</span>)</span>.</span>contains</span>(</span>key.</span>stringValue</span>)</span></span>
    }</span></span>
  }</span></span>
</span>
  struct</span> DynamicCodingKeys</span>:</span> CodingKey</span> {</span></span>
    var</span> stringValue:</span> String</span></span>
    init?</span>(</span>stringValue</span>: </span>String</span>)</span> {</span></span>
      self</span>.</span>stringValue</span> =</span> stringValue</span></span>
    }</span></span>
</span>
    //</span> not used here, but a protocol requirement</span></span>
    var</span> intValue:</span> Int</span>?</span></span>
    init?</span>(</span>intValue</span>: </span>Int</span>)</span> {</span></span>
      return</span> nil</span></span>
    }</span></span>
  }</span></span>
</span>
  init</span>(</span>from</span> decoder</span>: </span>Decoder</span>)</span> throws</span> {</span></span>
    let</span> container </span>=</span> try</span> decoder.</span>container</span>(</span>keyedBy</span>:</span> KnownCodingKeys.</span>self</span>)</span></span>
    self</span>.</span>a</span> =</span> try</span> container.</span>decode</span>(</span>Int</span>.</span>self</span>, </span>forKey</span>:</span> .</span>a</span>)</span></span>
    self</span>.</span>b</span> =</span> try</span> container.</span>decode</span>(</span>String</span>.</span>self</span>, </span>forKey</span>:</span> .</span>b</span>)</span></span>
</span>
    self</span>.</span>extraOptions</span> =</span> [</span>:</span>]</span></span>
    let</span> extraContainer </span>=</span> try</span> decoder.</span>container</span>(</span>keyedBy</span>:</span> DynamicCodingKeys.</span>self</span>)</span></span>
</span>
    for</span> key </span>in</span> extraContainer.allKeys </span>where</span> KnownCodingKeys.</span>doesNotContain</span>(</span>key</span>)</span> {</span></span>
      let</span> decoded </span>=</span> try</span> extraContainer.</span>decode</span>(</span>AnyDecodable.</span>self</span>, </span>forKey</span>:</span> DynamicCodingKey</span>(</span>stringValue</span>:</span> key.</span>stringValue</span>)</span>!</span>)</span></span>
      self</span>.</span>extraOptions</span>[</span>key.</span>stringValue</span>]</span> =</span> decoded</span></span>
    }</span></span>
  }</span></span>
}</span></span></code></pre>
The static function KnownCodingKeys.doesNotContain(_:)</code> is totally not necessary, but it makes the loop in the initializer nicer to read.</p>
When trying to run this, we see that it works as intended!</p>
let</span> json </span>=</span> """</span></span>
{</span></span>
  "a": 1,</span></span>
  "b": "foobar",</span></span>
  "option.extra": {</span></span>
    "a": 1</span></span>
  },</span></span>
  "option.extra.more": 4</span></span>
}</span></span>
"""</span>.</span>data</span>(</span>using</span>:</span> .</span>utf8</span>)</span>!</span></span>
</span>
let</span> decoded </span>=</span> try</span> JSONDecoder</span>(</span>)</span>.</span>decode</span>(</span>Json.</span>self</span>, </span>from</span>:</span> json</span>)</span></span>
print</span>(</span>decoded</span>)</span></span></code></pre>Json</span>(</span></span>
  a</span>:</span> 1</span>,</span></span>
  b</span>:</span> "</span>foobar</span>"</span>,</span></span>
  extraOptions</span>:</span> [</span></span>
    "</span>option.extra</span>"</span>:</span> AnyDecodable</span>(</span>[</span>"</span>a</span>"</span>:</span> 1</span>]</span>)</span>,</span></span>
    "</span>option.extra.more</span>"</span>:</span> AnyDecodable</span>(</span>4</span>)</span></span>
  ]</span></span>
)</span></span></code></pre>
That's it, cya for the next blog post in - check's watch</em> - three years 😅</p>



Docker Swift Runtime
2018-01-29T12:25:00+01:00
I recently wrote a small Vapor</a> web app in Swift which I then wanted to deploy on my server inside a docker container. Not having all too much experience with docker my simple Dockerfile looked something like this.</p>
FROM</span> swift:4</span></span>
</span>
COPY</span> .</span> .</span></span>
</span>
EXPOSE</span> 8080</span></span>
</span>
RUN</span> swift</span> build</span> -</span>-configuration</span> release</span></span>
</span>
ENTRYPOINT</span> [</span>"</span>.build/release/Run</span>"</span>]</span></span>
CMD</span> [</span>"</span>serve</span>"</span>,</span> "</span>--env=production</span>"</span>]</span></span></code></pre>
Whilst this worked perfectly fine, the image size turned out to be a whopping 1.5GB. That didn't quite feel right if for an app as small as mine.</p>
The problem is of course that the swift</code> image contains the entire development environment that we don't actually need anymore after having built the app.</p>
After trying to go the route of automating the build process by using a container dedicated to building the app then extracting everything I needed from that container and building a new one from that I saw that docker actually had this functionality built in. It's called a multistage build and lets you use separate images to perform whatever it is you want to do. In this case use an image for compiling the app and a separate one to actually package up and run the app binary and configs.</p>
The Dockerfile for that case doesn't get a lot more complicated, we just set a second base image after we're done with the first one and copy over the necessary files.</p>
FROM</span> swift:4</span> as</span> build</span></span>
</span>
COPY</span> .</span> .</span></span>
</span>
RUN</span> mkdir</span> -</span>p</span> /build/lib</span> &&</span> cp</span> -</span>R</span> /usr/lib/swift/linux/</span>*</span>.so</span> /build/lib</span></span>
RUN</span> swift</span> build</span> -</span>c</span> release</span> &&</span> mv</span> `</span>swift</span> build</span> -</span>c</span> release</span> -</span>-show-bin-path</span>`</span> /build/bin</span></span>
</span>
FROM</span> ubuntu:16.04</span></span>
</span>
RUN</span> apt-get</span> -</span>q</span> update</span> &&</span> \</span></span>
    apt-get</span> -</span>q</span> install</span> -</span>y</span> \</span></span>
    libatomic1</span> \</span></span>
    libbsd0</span> \</span></span>
    libcurl3</span> \</span></span>
    libicu55</span> \</span></span>
    libxml2</span> \</span></span>
    &&</span> rm</span> -</span>r</span> /var/lib/apt/lists/</span>*</span></span>
</span>
COPY</span> -</span>-from=build</span> /build/bin</span> .</span></span>
COPY</span> -</span>-from=build</span> /build/lib/</span>*</span> /usr/lib</span></span>
COPY</span> -</span>-from=build</span> /Config</span> ./Config</span></span>
</span>
EXPOSE</span> 8080</span></span>
</span>
ENTRYPOINT[</span>"</span>./release/Run</span>"</span>]</span></span>
CMD[</span>"</span>serve</span>"</span>,</span> "</span>--env=production</span>"</span>]</span></span></code></pre>
To properly set up the second image to run our app we just have to install a few dependencies and copy over the contents of /usr/lib/swift/...</code>. The rest is the same as before.</p>
Since we now no longer store all dev dependencies in our image, we've just made our image ~1.3GB smaller (around 200MB), that's quite the improvement!</p>
To wrap things up even nicer I published the runtime image as kiliankoe/swift-runtime</code></a> so that we can reduce our Dockerfile to the following.</p>
FROM</span> swift:4</span> as</span> build</span></span>
</span>
COPY</span> .</span> .</span></span>
</span>
RUN</span> swift</span> build</span> -</span>c</span> release</span> &&</span> mv</span> `</span>swift</span> build</span> -</span>c</span> release</span> -</span>-show-bin-path</span>`</span> /build/bin</span></span>
</span>
FROM</span> kiliankoe/swift-runtime:latest</span></span>
</span>
COPY</span> -</span>-from=build</span> /build/bin</span> .</span></span>
COPY</span> -</span>-from=build</span> /Config</span> ./Config</span></span>
</span>
EXPOSE</span> 8080</span></span>
</span>
ENTRYPOINT[</span>"</span>./release/Run</span>"</span>]</span></span>
CMD[</span>"</span>serve</span>"</span>,</span> "</span>--env=production</span>"</span>]</span></span></code></pre>
My image above is still based on ubuntu:16.04</code>, which definitely shouldn't be a necessity and accounts for most of the remaining 200MB. It would be nice to have far slimmer images in the future, based on alpine for example, but for time being the efforts needed seem to be insurmountable</a>.</p>