Decoding Unknown Data with Codable

2021-09-01T00:00:00+00:00
A friend recently asked me how to decode the following JSON through Swift's Codable</code> type.</p>
{</span></span>
  "</span>a</span>"</span>: </span>1</span>,</span></span>
  "</span>b</span>"</span>: </span>"</span>foobar</span>"</span>,</span></span>
  "</span>option.extra</span>"</span>:</span> {</span></span>
    "</span>a</span>"</span>:</span> 1</span></span>
  }</span>,</span></span>
  "</span>option.extra.more</span>"</span>: </span>4</span></span>
}</span></span></code></pre>
The problem is that only the first two keys (a</code> and b</code>) are guaranteed to exist, anything aside from that (here everything prefixed with option</code>) is extra and unknown at compile-time. Unknown keys should not be discarded however, they should be decoded into a dictionary for later use, we want a type like this.</p>
struct</span> Json</span> {</span></span>
  var</span> a:</span> Int</span></span>
  var</span> b:</span> String</span></span>
  var</span> extraOptions:</span> [</span>String</span>:</span> ?</span>]</span></span>
}</span></span></code></pre>
It's a bit unclear what type to use for extraOptions</code>, since we don't really know what the values are. They can be anything, as long as it's decodable. The folks at Flight School</a> have a fantastic package for us to facilitate that conveniently called AnyCodable</a>, which provides type-erased wrappers for Encodable</code>, Decodable</code>, and Codable</code> values. We can use that to represent the requirement of having anything</em>, as long as it's decodable, by defining the type as follows.</p>
struct</span> Json</span> {</span></span>
  var</span> a:</span> Int</span></span>
  var</span> b:</span> String</span></span>
  var</span> extraOptions:</span> [</span>String</span>:</span> AnyDecodable</span>]</span></span>
}</span></span></code></pre>
As it stands we're getting the automatically synthesized implementation of Json</code>'s initializer, which will try to find a dictionary called extraOptions</code> in the source JSON, which obviously isn't present. We'll have to implement the initializer ourselves. Doing so will also require us to define a type conforming to CodingKey</code>, which we can use to specify the encoded keys in our source data.</p>
extension</span> Json</span>:</span> Decodable</span> {</span></span>
  private</span> enum</span> CodingKeys</span>:</span> CodingKey</span> {</span></span>
    case</span> a</span>, </span>b</span></span>
  }</span></span>
</span>
  init</span>(</span>from</span> decoder</span>: </span>Decoder</span>)</span> throws</span> {</span></span>
    let</span> container </span>=</span> try</span> decoder.</span>container</span>(</span>keyedBy</span>:</span> CodingKeys.</span>self</span>)</span></span>
    self</span>.</span>a</span> =</span> try</span> container.</span>decode</span>(</span>Int</span>.</span>self</span>, </span>forKey</span>:</span> .</span>a</span>)</span></span>
    self</span>.</span>b</span> =</span> try</span> container.</span>decode</span>(</span>String</span>.</span>self</span>, </span>forKey</span>:</span> .</span>b</span>)</span></span>
    self</span>.</span>extraOptions</span> =</span> [</span>:</span>]</span></span>
  }</span></span>
}</span></span></code></pre>
We've now run into the problem of somehow having to specify all keys in our CodingKeys</code> enum, which obviously doesn't help for unknown keys. What we can do though is to use the CodingKey</code> protocol to create our own dynamic</em> key handling type, which can handle any String</code> as a key.</p>
struct</span> DynamicCodingKeys</span>:</span> CodingKey</span> {</span></span>
  var</span> stringValue:</span> String</span></span>
  init?</span>(</span>stringValue</span>: </span>String</span>)</span> {</span></span>
    self</span>.</span>stringValue</span> =</span> stringValue</span></span>
  }</span></span>
</span>
  //</span> not used here, but a protocol requirement</span></span>
  var</span> intValue:</span> Int</span>?</span></span>
  init?</span>(</span>intValue</span>: </span>Int</span>)</span> {</span></span>
    return</span> nil</span></span>
  }</span></span>
}</span></span></code></pre>
While we could use this entirely instead of our previous implementation, I personally find it quite a bit cleaner to keep the existing CodingKeys</code> for known keys, but cover everything else with the DynamicCodingKeys</code>. Creating a new container from our decoder keyed with DynamicCodingKeys</code> however will give us a container containing all</em> keys, not just the extra ones, so we'll have to filter out those we've already taken care of, leaving us with something like this.</p>
extension</span> Json</span>:</span> Decodable</span> {</span></span>
  private</span> enum</span> KnownCodingKeys</span>:</span> CodingKey</span>, </span>CaseIterable</span> {</span></span>
    case</span> a</span>, </span>b</span></span>
</span>
    static</span> func</span> doesNotContain</span>(</span>_</span> key</span>: </span>Json.DynamicCodingKeys</span>)</span> -></span> Bool</span> {</span></span>
      !</span>Self</span>.</span>allCases</span>.</span>map</span>(</span>\.</span>stringValue</span>)</span>.</span>contains</span>(</span>key.</span>stringValue</span>)</span></span>
    }</span></span>
  }</span></span>
</span>
  struct</span> DynamicCodingKeys</span>:</span> CodingKey</span> {</span></span>
    var</span> stringValue:</span> String</span></span>
    init?</span>(</span>stringValue</span>: </span>String</span>)</span> {</span></span>
      self</span>.</span>stringValue</span> =</span> stringValue</span></span>
    }</span></span>
</span>
    //</span> not used here, but a protocol requirement</span></span>
    var</span> intValue:</span> Int</span>?</span></span>
    init?</span>(</span>intValue</span>: </span>Int</span>)</span> {</span></span>
      return</span> nil</span></span>
    }</span></span>
  }</span></span>
</span>
  init</span>(</span>from</span> decoder</span>: </span>Decoder</span>)</span> throws</span> {</span></span>
    let</span> container </span>=</span> try</span> decoder.</span>container</span>(</span>keyedBy</span>:</span> KnownCodingKeys.</span>self</span>)</span></span>
    self</span>.</span>a</span> =</span> try</span> container.</span>decode</span>(</span>Int</span>.</span>self</span>, </span>forKey</span>:</span> .</span>a</span>)</span></span>
    self</span>.</span>b</span> =</span> try</span> container.</span>decode</span>(</span>String</span>.</span>self</span>, </span>forKey</span>:</span> .</span>b</span>)</span></span>
</span>
    self</span>.</span>extraOptions</span> =</span> [</span>:</span>]</span></span>
    let</span> extraContainer </span>=</span> try</span> decoder.</span>container</span>(</span>keyedBy</span>:</span> DynamicCodingKeys.</span>self</span>)</span></span>
</span>
    for</span> key </span>in</span> extraContainer.allKeys </span>where</span> KnownCodingKeys.</span>doesNotContain</span>(</span>key</span>)</span> {</span></span>
      let</span> decoded </span>=</span> try</span> extraContainer.</span>decode</span>(</span>AnyDecodable.</span>self</span>, </span>forKey</span>:</span> DynamicCodingKey</span>(</span>stringValue</span>:</span> key.</span>stringValue</span>)</span>!</span>)</span></span>
      self</span>.</span>extraOptions</span>[</span>key.</span>stringValue</span>]</span> =</span> decoded</span></span>
    }</span></span>
  }</span></span>
}</span></span></code></pre>
The static function KnownCodingKeys.doesNotContain(_:)</code> is totally not necessary, but it makes the loop in the initializer nicer to read.</p>
When trying to run this, we see that it works as intended!</p>
let</span> json </span>=</span> """</span></span>
{</span></span>
  "a": 1,</span></span>
  "b": "foobar",</span></span>
  "option.extra": {</span></span>
    "a": 1</span></span>
  },</span></span>
  "option.extra.more": 4</span></span>
}</span></span>
"""</span>.</span>data</span>(</span>using</span>:</span> .</span>utf8</span>)</span>!</span></span>
</span>
let</span> decoded </span>=</span> try</span> JSONDecoder</span>(</span>)</span>.</span>decode</span>(</span>Json.</span>self</span>, </span>from</span>:</span> json</span>)</span></span>
print</span>(</span>decoded</span>)</span></span></code></pre>Json</span>(</span></span>
  a</span>:</span> 1</span>,</span></span>
  b</span>:</span> "</span>foobar</span>"</span>,</span></span>
  extraOptions</span>:</span> [</span></span>
    "</span>option.extra</span>"</span>:</span> AnyDecodable</span>(</span>[</span>"</span>a</span>"</span>:</span> 1</span>]</span>)</span>,</span></span>
    "</span>option.extra.more</span>"</span>:</span> AnyDecodable</span>(</span>4</span>)</span></span>
  ]</span></span>
)</span></span></code></pre>
That's it, cya for the next blog post in - check's watch</em> - three years 😅</p>
kilko.de - json

Decoding Unknown Data with Codable
