Jekyll2024-01-04T14:42:07+00:00https://sergey-melnychuk.github.io//$ cat /dev/blogWrite an awesome description for your new site here. You can edit this line in _config.yml. It will appear in your document head meta (for Google search results) and in your feed.xml site description.Sergey MelnychukLearning ECDSA with arkworks-rs2023-11-22T18:20:42+00:002023-11-22T18:20:42+00:00https://sergey-melnychuk.github.io/2023/11/22/rust-arkworks-ecdsa<p>While I was <a href="https://github.com/sergey-melnychuk/doing-some-blockchain">doing-some-blockchain</a> recently, I have found out that Elliptic Curve algebra does not seem to fit easily into <code class="language-plaintext highlighter-rouge">u128</code>, so this is a second attepmt, now done properly, using <a href="https://github.com/arkworks-rs">arkworks-rs</a>.</p>
<p>TL/DR: <a href="https://github.com/sergey-melnychuk/doing-some-curves">code</a>.</p>
<p>Note: this is a purely educational example, meant just to show how abstract math can be implemented in real world using propert tools & libraries. Do not use this implementation for anything beyound education. Consider yourself warned.</p>
<p>For a warmup let’s implement <a href="https://en.wikipedia.org/wiki/Diffie–Hellman_key_exchange">DHKE</a> on a <code class="language-plaintext highlighter-rouge">secp256k1</code> elliptic curve (the one used in Bitcoin):</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// cargo add rand ark-std ark-ff ark-secp256k1</span>
<span class="nd">#[test]</span>
<span class="k">fn</span> <span class="nf">test_dhke</span><span class="p">()</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">ark_std</span><span class="p">::{</span><span class="n">rand</span><span class="p">,</span> <span class="n">UniformRand</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">ark_secp256k1</span><span class="p">::{</span><span class="n">Affine</span><span class="p">,</span> <span class="n">Fr</span> <span class="k">as</span> <span class="n">F</span><span class="p">,</span> <span class="n">G_GENERATOR_X</span><span class="p">,</span> <span class="n">G_GENERATOR_Y</span><span class="p">};</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">rng</span> <span class="o">=</span> <span class="nn">rand</span><span class="p">::</span><span class="nf">thread_rng</span><span class="p">();</span>
<span class="k">let</span> <span class="n">g</span> <span class="o">=</span> <span class="nn">Affine</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">G_GENERATOR_X</span><span class="p">,</span> <span class="n">G_GENERATOR_Y</span><span class="p">);</span>
<span class="k">let</span> <span class="n">a</span> <span class="o">=</span> <span class="nn">F</span><span class="p">::</span><span class="nf">rand</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">rng</span><span class="p">);</span>
<span class="k">let</span> <span class="n">ga</span> <span class="o">=</span> <span class="n">g</span> <span class="o">*</span> <span class="n">a</span><span class="p">;</span>
<span class="k">let</span> <span class="n">b</span> <span class="o">=</span> <span class="nn">F</span><span class="p">::</span><span class="nf">rand</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">rng</span><span class="p">);</span>
<span class="k">let</span> <span class="n">gb</span> <span class="o">=</span> <span class="n">g</span> <span class="o">*</span> <span class="n">b</span><span class="p">;</span>
<span class="k">let</span> <span class="n">one</span> <span class="o">=</span> <span class="n">ga</span> <span class="o">+</span> <span class="n">gb</span><span class="p">;</span>
<span class="k">let</span> <span class="n">two</span> <span class="o">=</span> <span class="n">gb</span> <span class="o">+</span> <span class="n">ga</span><span class="p">;</span>
<span class="c">// shared secrets must match</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">one</span><span class="p">,</span> <span class="n">two</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cargo test
...
running 1 test
test tests::test_dhke ... ok
</code></pre></div></div>
<p>Now <a href="https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm">ECDSA</a>: first add necessary dependencies:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cargo add sha2 hex
cargo add ark-serialize --features derive
</code></pre></div></div>
<p>So that final dependencies list looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[dependencies]
ark-ff = "0.4.2"
ark-secp256k1 = "0.4.0"
ark-serialize = { version = "0.4.2", features = ["derive"] }
ark-std = "0.4.0"
hex = "0.4.3"
rand = "0.8.5"
sha2 = "0.10.8"
</code></pre></div></div>
<p>First step is to derive a Public Key (PK) from a Secret Key (SK) - with SK being a scalar (just a regular number, not yet a point on the curve), it is enough to just multiply curve’s generator point (G) by the scalar to get a point on the curve. Thanks to the <a href="https://en.wikipedia.org/wiki/Discrete_logarithm">Discrete Logarithm problem</a>, inversing this operation (and thus “cracking” a secret key from a public one) is considered extremely computationally intense (for the numbers of a certain size, say 256 bits).</p>
<p>For simplisity the API I’m presenting is going to use raw bytes slices and vectors to represent both public and secret keys.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">get_pk</span><span class="p">(</span><span class="n">sk</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">])</span> <span class="k">-></span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="c">// Derive Public Key from a Secret Key:</span>
<span class="c">// </span>
<span class="c">// SK - secret key (scalar)</span>
<span class="c">// G - generator point</span>
<span class="c">// </span>
<span class="c">// PK = G * SK</span>
<span class="k">let</span> <span class="n">g</span> <span class="o">=</span> <span class="nn">Affine</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">G_GENERATOR_X</span><span class="p">,</span> <span class="n">G_GENERATOR_Y</span><span class="p">);</span>
<span class="k">let</span> <span class="n">sk</span> <span class="o">=</span> <span class="nn">F</span><span class="p">::</span><span class="nf">from_be_bytes_mod_order</span><span class="p">(</span><span class="o">&</span><span class="n">sk</span><span class="p">);</span>
<span class="k">let</span> <span class="n">pk</span> <span class="o">=</span> <span class="n">g</span> <span class="o">*</span> <span class="n">sk</span><span class="p">;</span>
<span class="nf">into_bytes</span><span class="p">(</span><span class="o">&</span><span class="n">pk</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The signing part, pretty straightforward math:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">sig</span><span class="p">(</span><span class="n">sk</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">],</span> <span class="n">msg</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">])</span> <span class="k">-></span> <span class="p">(</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span><span class="p">,</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span><span class="p">)</span> <span class="p">{</span>
<span class="c">// Signing:</span>
<span class="c">// </span>
<span class="c">// SK - secret key</span>
<span class="c">// PK - public key (PK = SK * G)</span>
<span class="c">// m - message</span>
<span class="c">// H - sha256</span>
<span class="c">// (r, s) - signature</span>
<span class="c">// </span>
<span class="c">// k = H(H(SK) || H(m))</span>
<span class="c">// R = k * G</span>
<span class="c">// r = R.x</span>
<span class="c">// s = k' * (h + r * SK)</span>
<span class="k">let</span> <span class="n">k</span> <span class="o">=</span> <span class="nf">hash</span><span class="p">(</span><span class="o">&</span><span class="p">[</span><span class="o">&</span><span class="nf">hash</span><span class="p">(</span><span class="o">&</span><span class="p">[</span><span class="n">sk</span><span class="p">]),</span> <span class="o">&</span><span class="nf">hash</span><span class="p">(</span><span class="o">&</span><span class="p">[</span><span class="n">msg</span><span class="p">])]);</span>
<span class="k">let</span> <span class="n">k</span> <span class="o">=</span> <span class="nn">F</span><span class="p">::</span><span class="nf">from_be_bytes_mod_order</span><span class="p">(</span><span class="o">&</span><span class="n">k</span><span class="p">);</span>
<span class="k">let</span> <span class="n">ki</span> <span class="o">=</span> <span class="n">k</span><span class="nf">.inverse</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="k">let</span> <span class="n">h</span> <span class="o">=</span> <span class="nf">hash</span><span class="p">(</span><span class="o">&</span><span class="p">[</span><span class="n">msg</span><span class="p">]);</span>
<span class="k">let</span> <span class="n">h</span> <span class="o">=</span> <span class="nn">F</span><span class="p">::</span><span class="nf">from_be_bytes_mod_order</span><span class="p">(</span><span class="o">&</span><span class="n">h</span><span class="p">);</span>
<span class="k">let</span> <span class="n">sk</span> <span class="o">=</span> <span class="nn">F</span><span class="p">::</span><span class="nf">from_be_bytes_mod_order</span><span class="p">(</span><span class="o">&</span><span class="n">sk</span><span class="p">);</span>
<span class="k">let</span> <span class="n">g</span> <span class="o">=</span> <span class="nn">Affine</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">G_GENERATOR_X</span><span class="p">,</span> <span class="n">G_GENERATOR_Y</span><span class="p">);</span>
<span class="k">let</span> <span class="n">r</span> <span class="o">=</span> <span class="n">g</span> <span class="o">*</span> <span class="n">k</span><span class="p">;</span>
<span class="k">let</span> <span class="n">r</span> <span class="o">=</span> <span class="nn">Affine</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">r</span><span class="p">);</span>
<span class="k">let</span> <span class="n">rx</span> <span class="o">=</span> <span class="nn">F</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">r</span><span class="py">.x</span><span class="nf">.into_bigint</span><span class="p">());</span>
<span class="k">let</span> <span class="n">s</span> <span class="o">=</span> <span class="n">ki</span> <span class="o">*</span> <span class="p">(</span><span class="n">h</span> <span class="o">+</span> <span class="n">rx</span> <span class="o">*</span> <span class="n">sk</span><span class="p">);</span>
<span class="p">(</span><span class="nf">into_bytes</span><span class="p">(</span><span class="o">&</span><span class="n">rx</span><span class="p">),</span> <span class="nf">into_bytes</span><span class="p">(</span><span class="o">&</span><span class="n">s</span><span class="p">))</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The signature verification part: the whole “trick” is that <code class="language-plaintext highlighter-rouge">(h + r * SK)</code> and it’s modular inverse cancel each other out, so the original <code class="language-plaintext highlighter-rouge">R</code> is restored and compared to provided with the signature one (x coordinates really).</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">ver</span><span class="p">(</span><span class="n">pk</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">],</span> <span class="n">msg</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">],</span> <span class="n">sig</span><span class="p">:</span> <span class="p">(</span><span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">],</span> <span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">]))</span> <span class="k">-></span> <span class="nb">bool</span> <span class="p">{</span>
<span class="c">// Verification</span>
<span class="c">// </span>
<span class="c">// R = (h * s') * G + (r * s') * PK</span>
<span class="c">// For a valid signature: R.x == sig.r</span>
<span class="c">//</span>
<span class="c">// R = (h * s') * G + (r * s') * PK</span>
<span class="c">// R = (h * s') * G + (r * s') * SK * G</span>
<span class="c">// R = (h + r * SK) * s' * G</span>
<span class="c">// R = (h + r * SK) * (k' * (h + r * SK))' * G</span>
<span class="c">// R = (h + r * SK) * k * (h + r * SK)' * G</span>
<span class="c">// R = k * G</span>
<span class="k">let</span> <span class="n">pk</span><span class="p">:</span> <span class="n">Projective</span> <span class="o">=</span> <span class="nf">from_bytes</span><span class="p">(</span><span class="o">&</span><span class="n">pk</span><span class="p">);</span>
<span class="k">let</span> <span class="p">(</span><span class="n">rx</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> <span class="o">=</span> <span class="n">sig</span><span class="p">;</span>
<span class="k">let</span> <span class="n">rx</span><span class="p">:</span> <span class="nn">types</span><span class="p">::</span><span class="n">Number</span> <span class="o">=</span> <span class="nf">from_bytes</span><span class="p">(</span><span class="n">rx</span><span class="p">);</span>
<span class="k">let</span> <span class="n">s</span><span class="p">:</span> <span class="nn">types</span><span class="p">::</span><span class="n">Number</span> <span class="o">=</span> <span class="nf">from_bytes</span><span class="p">(</span><span class="n">s</span><span class="p">);</span>
<span class="k">let</span> <span class="n">si</span> <span class="o">=</span> <span class="n">s</span><span class="nf">.inverse</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="k">let</span> <span class="n">h</span> <span class="o">=</span> <span class="nf">hash</span><span class="p">(</span><span class="o">&</span><span class="p">[</span><span class="n">msg</span><span class="p">]);</span>
<span class="k">let</span> <span class="n">h</span> <span class="o">=</span> <span class="nn">F</span><span class="p">::</span><span class="nf">from_be_bytes_mod_order</span><span class="p">(</span><span class="o">&</span><span class="n">h</span><span class="p">);</span>
<span class="k">let</span> <span class="n">g</span> <span class="o">=</span> <span class="nn">Affine</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">G_GENERATOR_X</span><span class="p">,</span> <span class="n">G_GENERATOR_Y</span><span class="p">);</span>
<span class="k">let</span> <span class="n">r</span> <span class="o">=</span> <span class="n">g</span> <span class="o">*</span> <span class="n">h</span> <span class="o">*</span> <span class="n">si</span> <span class="o">+</span> <span class="n">pk</span> <span class="o">*</span> <span class="n">rx</span> <span class="o">*</span> <span class="n">si</span><span class="p">;</span>
<span class="k">let</span> <span class="n">r</span> <span class="o">=</span> <span class="nn">Affine</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">r</span><span class="p">);</span>
<span class="n">rx</span> <span class="o">==</span> <span class="nn">F</span><span class="p">::</span><span class="nf">from</span><span class="p">(</span><span class="n">r</span><span class="py">.x</span><span class="nf">.into_bigint</span><span class="p">())</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Does it work? Checking:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[test]</span>
<span class="k">fn</span> <span class="nf">test_ecdsa</span><span class="p">()</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">super</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="k">let</span> <span class="n">sk</span> <span class="o">=</span> <span class="s">"43cdf7c47a34cac01e717ad098bde292c2b3972719da38b7d38706be25706d4f"</span><span class="p">;</span>
<span class="k">let</span> <span class="n">sk</span> <span class="o">=</span> <span class="nn">hex</span><span class="p">::</span><span class="nf">decode</span><span class="p">(</span><span class="n">sk</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="k">let</span> <span class="n">pk</span> <span class="o">=</span> <span class="nf">get_pk</span><span class="p">(</span><span class="o">&</span><span class="n">sk</span><span class="p">);</span>
<span class="k">let</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">b</span><span class="s">"the quick brown fox jumps over the lazy dog"</span><span class="p">;</span>
<span class="k">let</span> <span class="p">(</span><span class="n">r</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> <span class="o">=</span> <span class="nf">sig</span><span class="p">(</span><span class="o">&</span><span class="n">sk</span><span class="p">,</span> <span class="n">msg</span><span class="p">);</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="nf">ver</span><span class="p">(</span><span class="o">&</span><span class="n">pk</span><span class="p">,</span> <span class="n">msg</span><span class="p">,</span> <span class="p">(</span><span class="o">&</span><span class="n">r</span><span class="p">,</span> <span class="o">&</span><span class="n">s</span><span class="p">)));</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Seems like it does:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cargo test
...
running 2 tests
test tests::test_dhke ... ok
test tests::test_ecdsa ... ok
...
</code></pre></div></div>
<p>To test against random secret keys:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[test]</span>
<span class="k">fn</span> <span class="nf">test_ecdsa</span><span class="p">()</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">super</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">ark_std</span><span class="p">::{</span><span class="n">rand</span><span class="p">,</span> <span class="n">UniformRand</span><span class="p">};</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">rng</span> <span class="o">=</span> <span class="nn">rand</span><span class="p">::</span><span class="nf">thread_rng</span><span class="p">();</span>
<span class="k">let</span> <span class="n">a</span> <span class="o">=</span> <span class="nn">F</span><span class="p">::</span><span class="nf">rand</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">rng</span><span class="p">);</span>
<span class="k">let</span> <span class="n">sk</span> <span class="o">=</span> <span class="nf">into_bytes</span><span class="p">(</span><span class="o">&</span><span class="n">a</span><span class="p">);</span>
<span class="c">// let sk = "43cdf7c47a34cac01e717ad098bde292c2b3972719da38b7d38706be25706d4f";</span>
<span class="c">// let sk = hex::decode(sk).unwrap();</span>
<span class="k">let</span> <span class="n">pk</span> <span class="o">=</span> <span class="nf">get_pk</span><span class="p">(</span><span class="o">&</span><span class="n">sk</span><span class="p">);</span>
<span class="k">let</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">b</span><span class="s">"the quick brown fox jumps over the lazy dog"</span><span class="p">;</span>
<span class="k">let</span> <span class="p">(</span><span class="n">r</span><span class="p">,</span> <span class="n">s</span><span class="p">)</span> <span class="o">=</span> <span class="nf">sig</span><span class="p">(</span><span class="o">&</span><span class="n">sk</span><span class="p">,</span> <span class="n">msg</span><span class="p">);</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="nf">ver</span><span class="p">(</span><span class="o">&</span><span class="n">pk</span><span class="p">,</span> <span class="n">msg</span><span class="p">,</span> <span class="p">(</span><span class="o">&</span><span class="n">r</span><span class="p">,</span> <span class="o">&</span><span class="n">s</span><span class="p">)));</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Full code is on <a href="https://github.com/sergey-melnychuk/doing-some-curves">GitHub</a>. Implementations of <code class="language-plaintext highlighter-rouge">{from, into}_bytes</code> and <code class="language-plaintext highlighter-rouge">hash</code> are pretty straightforward, so not providing them here for brevity. They are present in the repository of course.</p>
<p>ONE NOTE THOUGH:</p>
<p>There is one tricky moment in this implementation. Attentive reader might have noticed weird “flipping” (serializing to bytes and then deserializing back to point) of <code class="language-plaintext highlighter-rouge">R</code> happening in both signing and verification parts (<code class="language-plaintext highlighter-rouge">let r: Projective = from_bytes(&into_bytes(&r));</code>). The problem I had without such “flip”, was that the original point <code class="language-plaintext highlighter-rouge">R</code> was restored (byte representation did match the original one), but <code class="language-plaintext highlighter-rouge">x</code> coordinate did not match the original <code class="language-plaintext highlighter-rouge">R</code>’s <code class="language-plaintext highlighter-rouge">x</code> coordinate, for reasons I don’t quite understand. Pretty sure I must have misused <code class="language-plaintext highlighter-rouge">arkworks-rs</code>, and there is a proper way to express it, but I don’t know it yet! Leaving this as an excercise for a reader :) As long as tests pass, I think the educational purposes of this example implementation are achieved.</p>
<p>UPDATE: ‘ONE NOTE’ FIX</p>
<p><code class="language-plaintext highlighter-rouge">Affine::from</code> was necessary to convert <code class="language-plaintext highlighter-rouge">R</code> from projective point to an affine one! Thank you Bayram!</p>
<p>References:</p>
<ul>
<li><a href="https://medium.com/@prajwolgyawali/getting-started-with-arkworks-rs-e5ceaca895a9">Getting started with arkworks-rs</a></li>
<li><a href="https://ventral.digital/posts/2023/8/22/applied-elliptic-curve-cryptography">Applied Elliptic Curve Cryptography</a></li>
</ul>Sergey MelnychukWhile I was doing-some-blockchain recently, I have found out that Elliptic Curve algebra does not seem to fit easily into u128, so this is a second attepmt, now done properly, using arkworks-rs.Get rid of typos with typos2023-11-20T07:55:42+00:002023-11-20T07:55:42+00:00https://sergey-melnychuk.github.io/2023/11/20/get-rid-of-typos<p>Typos in code and comments can be annoying. But they don’t have to be, with <a href="https://github.com/crate-ci/typos"><code class="language-plaintext highlighter-rouge">typos</code></a> it is super easy to introduce automated typos checking (and fixing).</p>
<p>Easy steps to get rid of typos. Once and for all.</p>
<ol>
<li>Install <code class="language-plaintext highlighter-rouge">typos</code> with <code class="language-plaintext highlighter-rouge">cargo install typos-cli</code> or <a href="https://github.com/crate-ci/typos#install">any other way</a>.</li>
<li>Run <code class="language-plaintext highlighter-rouge">typos</code> to detect.</li>
<li>Run <code class="language-plaintext highlighter-rouge">typos -w</code> to fix.</li>
<li>???</li>
<li>PROFIT!</li>
</ol>
<p>Maybe add a <a href="https://github.com/crate-ci/typos/blob/master/docs/github-action.md">Github Action</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>name: ...
on: [pull_request]
jobs:
...
typos:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: crate-ci/typos@v1.16.23
with:
files: .
</code></pre></div></div>
<p>That’s all folks!</p>Sergey MelnychukTypos in code and comments can be annoying. But they don’t have to be, with typos it is super easy to introduce automated typos checking (and fixing).Recursive SQL queries over tree2023-11-17T20:32:42+00:002023-11-17T20:32:42+00:00https://sergey-melnychuk.github.io/2023/11/17/recursive-sql-queries<p>Some 10+ years ago on a job interview I was asked to query the database for the comments tree in a front-end friendly way, meaning no heavy processing/sorting to happen on the back-end. Turns out this question was to check if I knew about the <a href="https://docs.oracle.com/cd/B13789_01/server.101/b10759/queries003.htm">Hierarchical Queries with Oracle</a> (<code class="language-plaintext highlighter-rouge">WITH RECURSIVE</code> on other databases), which I did not back then.</p>
<p>So this is how an idiomatic way to do it looks like, with SQLite.</p>
<p>Sample comment tree:</p>
<ul>
<li>(id=101) uno: A
<ul>
<li>(id=102) due: B
<ul>
<li>(id=108) tre: H</li>
<li>(id=110) uno: I</li>
</ul>
</li>
<li>(id=109) tre: J</li>
</ul>
</li>
<li>(id=103) due: C
<ul>
<li>(id=104): uno: D</li>
</ul>
</li>
<li>(id=105) tre: E
<ul>
<li>(id=106) uno: F
<ul>
<li>(id=107) due: G</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>Create and open SQLite database:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">touch </span>test.db
sqlite3 test.db
</code></pre></div></div>
<p>Define the table:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">comments</span> <span class="p">(</span>
<span class="n">entry_id</span> <span class="nb">SERIAL</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">,</span>
<span class="n">reply_to</span> <span class="nb">BIGINT</span> <span class="nb">UNSIGNED</span><span class="p">,</span>
<span class="n">issue_id</span> <span class="nb">BIGINT</span> <span class="nb">UNSIGNED</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">entry_datetime</span> <span class="nb">DATETIME</span> <span class="k">NOT</span> <span class="k">NULL</span> <span class="k">DEFAULT</span> <span class="k">CURRENT_TIMESTAMP</span><span class="p">,</span>
<span class="n">entry_content</span> <span class="nb">TEXT</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="n">entry_author</span> <span class="nb">TEXT</span> <span class="k">NOT</span> <span class="k">NULL</span><span class="p">,</span>
<span class="k">FOREIGN</span> <span class="k">KEY</span> <span class="p">(</span><span class="n">reply_to</span><span class="p">)</span> <span class="k">REFERENCES</span> <span class="n">comments</span><span class="p">(</span><span class="n">entry_id</span><span class="p">)</span>
<span class="p">);</span>
</code></pre></div></div>
<p>Insert comments:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">comments</span> <span class="p">(</span><span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">issue_id</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="mi">101</span><span class="p">,</span> <span class="k">NULL</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'uno'</span><span class="p">,</span> <span class="s1">'A'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">comments</span> <span class="p">(</span><span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">issue_id</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="mi">102</span><span class="p">,</span> <span class="mi">101</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'due'</span><span class="p">,</span> <span class="s1">'B'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">comments</span> <span class="p">(</span><span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">issue_id</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="mi">103</span><span class="p">,</span> <span class="k">NULL</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'due'</span><span class="p">,</span> <span class="s1">'C'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">comments</span> <span class="p">(</span><span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">issue_id</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="mi">104</span><span class="p">,</span> <span class="mi">103</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'uno'</span><span class="p">,</span> <span class="s1">'D'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">comments</span> <span class="p">(</span><span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">issue_id</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="mi">105</span><span class="p">,</span> <span class="k">NULL</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'tre'</span><span class="p">,</span> <span class="s1">'E'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">comments</span> <span class="p">(</span><span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">issue_id</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="mi">106</span><span class="p">,</span> <span class="mi">105</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'uno'</span><span class="p">,</span> <span class="s1">'F'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">comments</span> <span class="p">(</span><span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">issue_id</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="mi">107</span><span class="p">,</span> <span class="mi">106</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'due'</span><span class="p">,</span> <span class="s1">'G'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">comments</span> <span class="p">(</span><span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">issue_id</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="mi">108</span><span class="p">,</span> <span class="mi">102</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'due'</span><span class="p">,</span> <span class="s1">'H'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">comments</span> <span class="p">(</span><span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">issue_id</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="mi">109</span><span class="p">,</span> <span class="mi">101</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'tre'</span><span class="p">,</span> <span class="s1">'I'</span><span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">comments</span> <span class="p">(</span><span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">issue_id</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">)</span>
<span class="k">VALUES</span> <span class="p">(</span><span class="mi">110</span><span class="p">,</span> <span class="mi">102</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="s1">'uno'</span><span class="p">,</span> <span class="s1">'J'</span><span class="p">);</span>
</code></pre></div></div>
<p>Query the tree (pre-order traversal):</p>
<p><code class="language-plaintext highlighter-rouge">WITH RECURSIVE</code> is followed by a table-defining clause, that describes a starting rows set (<code class="language-plaintext highlighter-rouge">WHERE reply_to is NULL</code>) and a “connection” between tree layers with (<code class="language-plaintext highlighter-rouge">JOIN comments c ON c.reply_to = ct.entry_id</code>) and then glues them together using <code class="language-plaintext highlighter-rouge">UNION ALL</code>.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">WITH</span> <span class="k">RECURSIVE</span> <span class="n">comment_tree</span> <span class="p">(</span>
<span class="n">issue_id</span><span class="p">,</span>
<span class="n">entry_id</span><span class="p">,</span>
<span class="n">reply_to</span><span class="p">,</span>
<span class="n">entry_author</span><span class="p">,</span>
<span class="n">entry_content</span><span class="p">,</span>
<span class="n">depth</span>
<span class="p">)</span> <span class="k">AS</span> <span class="p">(</span>
<span class="k">SELECT</span>
<span class="n">issue_id</span><span class="p">,</span>
<span class="n">entry_id</span><span class="p">,</span>
<span class="n">reply_to</span><span class="p">,</span>
<span class="n">entry_author</span><span class="p">,</span>
<span class="n">entry_content</span><span class="p">,</span>
<span class="mi">0</span> <span class="k">AS</span> <span class="n">depth</span>
<span class="k">FROM</span> <span class="n">comments</span>
<span class="k">WHERE</span> <span class="n">reply_to</span> <span class="k">is</span> <span class="k">NULL</span>
<span class="k">UNION</span> <span class="k">ALL</span>
<span class="k">SELECT</span>
<span class="k">c</span><span class="p">.</span><span class="n">issue_id</span><span class="p">,</span>
<span class="k">c</span><span class="p">.</span><span class="n">entry_id</span><span class="p">,</span>
<span class="k">c</span><span class="p">.</span><span class="n">reply_to</span><span class="p">,</span>
<span class="k">c</span><span class="p">.</span><span class="n">entry_author</span><span class="p">,</span>
<span class="k">c</span><span class="p">.</span><span class="n">entry_content</span><span class="p">,</span>
<span class="n">ct</span><span class="p">.</span><span class="n">depth</span><span class="o">+</span><span class="mi">1</span> <span class="k">AS</span> <span class="n">depth</span>
<span class="k">FROM</span> <span class="n">comment_tree</span> <span class="n">ct</span>
<span class="k">JOIN</span> <span class="n">comments</span> <span class="k">c</span> <span class="k">ON</span> <span class="k">c</span><span class="p">.</span><span class="n">reply_to</span> <span class="o">=</span> <span class="n">ct</span><span class="p">.</span><span class="n">entry_id</span>
<span class="p">)</span>
<span class="k">SELECT</span> <span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span><span class="p">,</span> <span class="n">entry_author</span><span class="p">,</span> <span class="n">entry_content</span><span class="p">,</span> <span class="n">depth</span>
<span class="k">FROM</span> <span class="n">comment_tree</span>
<span class="k">WHERE</span> <span class="n">issue_id</span> <span class="o">=</span> <span class="mi">42</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">entry_id</span><span class="p">,</span> <span class="n">reply_to</span>
<span class="p">;</span>
</code></pre></div></div>
<p>Get the results (with <code class="language-plaintext highlighter-rouge">NULL</code> represented as <code class="language-plaintext highlighter-rouge">---</code> just to keep formatting straight):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>101|---|uno|A|0
102|101|due|B|1
103|---|due|C|0
104|103|uno|D|1
105|---|tre|E|0
106|105|uno|F|1
107|106|due|G|2
108|102|due|H|2
109|101|tre|I|1
110|102|uno|J|2
</code></pre></div></div>
<p>More details in the book: <a href="https://pragprog.com/titles/bksap1/sql-antipatterns-volume-1/">SQL Antipatterns, Volume 1</a>.</p>Sergey MelnychukSome 10+ years ago on a job interview I was asked to query the database for the comments tree in a front-end friendly way, meaning no heavy processing/sorting to happen on the back-end. Turns out this question was to check if I knew about the Hierarchical Queries with Oracle (WITH RECURSIVE on other databases), which I did not back then.What Actor Model is really about2023-05-25T20:42:42+00:002023-05-25T20:42:42+00:00https://sergey-melnychuk.github.io/2023/05/25/what-actor-model-is-about<p>TL/DR: read [1] and watch [2]:</p>
<ol>
<li><a href="https://arxiv.org/pdf/1008.1459">Actor Model of Computation</a></li>
<li><a href="https://www.youtube.com/watch?v=7erJ1DV_Tlo">The Actor Model (Hewitt, Meijer, Szyperski)</a></li>
</ol>
<p>During recent years “Actor Model [of computation]” (or simple “Actors”) turned into a buzzword. But as usually happens with buzzwords, more “buzz” leads to less understanding of what it is actually about. There are dosen of implementations of the Actor Model in a variety of languages, most of them usually even make sense and some of them are just masterpieces (Erlang VM, Akka). Heck, I even implemented one: <a href="https://github.com/sergey-melnychuk/uppercut/">Uppercut</a>!</p>
<p>In this post I will try to structure my understanding and findings about Actor Model as well as best (and worst) use-cases for it. To start with, I just must warn that “everything is an actor” is just freaking wrong! I strongly encourage you to read the <a href="https://arxiv.org/pdf/1008.1459">paper</a> and vatch the <a href="https://www.youtube.com/watch?v=7erJ1DV_Tlo">video</a> if you want to achieve deep understanding of the topic once and for all.</p>
<p>Not “Everything is an Actor”. Only Actor is an Actor. Message is not an Actor, it is a Message. Channel (the way messages travel between actors) is not an Actor, it is a Channel. Environment is not an actor… You got my point. So what is an Actor then?</p>
<blockquote>
<p>An Actor is a computational entity that, in response to a message it receives, can concurrently:</p>
<ul>
<li>send a finite number of messages to other Actors;</li>
<li>create a finite number of new Actors;</li>
<li>designate the behavior to be used for the next message it receives.</li>
</ul>
</blockquote>
<p>Note that there is no mention of memory, network, threads, mailboxes, dead-letter-queues, supervisors, “let it crash” and so on. Just message passing & message handling. Note also that there is no such concept as a “reply” from an Actor, all communications between Actors are async and one-way. The whole point of Actors is:</p>
<blockquote>
<p>All physically possible computation can be directly implemented using Actors.</p>
</blockquote>
<p>Simply put Actor model is about data-flow vs. control-flow. The control-flow is basically how modern hardware operates - there is an <code class="language-plaintext highlighter-rouge">IP</code> index that points to the next instruction. An (un)conditional jumps operate directly on this index to literally “control the flow” of execution. For example, a very simple branching instruction controls the flow to go either to address <code class="language-plaintext highlighter-rouge">[X+ N]</code> and call <code class="language-plaintext highlighter-rouge">do_this</code> function, or to address <code class="language-plaintext highlighter-rouge">[X+ M]</code> and call <code class="language-plaintext highlighter-rouge">do_that</code> function:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>if x == 1 { ## X+00: jne [X+ M]
do_this(); ## X+ N: call [do_this]
} else {
do_that(); ## X+ M: call [do_that]
}
</code></pre></div></div>
<p>Same logical result in Actor Model can be achieved by adjusting data-flow:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>handle(X): ## Actor message handler
if X == 1 {
this_actor ! DO_WORK(X) ## The work item is sent to `this_actor`
} else {
that_actor ! DO_WORK(X) ## The work item is sent to `that_actor`
}
</code></pre></div></div>
<p>“So, Can I implement <em>it</em> using Actors?” - yes, of course you can. And this is the problem. Actor Model is not a silver bullet, and in practice Actors need to be run on physical hardware that most likely implements <a href="https://en.wikipedia.org/wiki/Von_Neumann_architecture">Von Neumann architecture</a>. This means that (likely shared) memory, netowrk and threads need to be dealt with.</p>
<p>If I condense my experience with Actor Model to a single sentence, it will be</p>
<blockquote>
<p>Actor Model is a good fit for event-based solutions only.</p>
</blockquote>
<p>If a solution you’re building is event-based (as opposed to what I call state-based), the Actor Model allows scalable and robust way to use message-passing for event processing. Just use tools like <a href="https://doc.akka.io/docs/akka/current/stream/index.html">Akka Streams</a> to declaratively implement how exactly you want your events/messages processed and that’s pretty much it! <a href="https://docs.confluent.io/platform/current/streams/overview.html">Kafka Streams</a> is probably worth mentioning in this context as well.</p>
<p>From a concurrency standpoint, Actor is a synchronization primitive (Actor instance “owns” it’s state in a mutually exclusive way and does not allow sharing it directly), similar to a <code class="language-plaintext highlighter-rouge">Mutex</code>. So if you want to implement Actors directly (and not use tools like Akka Streams), you better know what your are doing, because there is a good chance you don’t.</p>
<p>If a business logic of a solution is better described in term of (relatively) small changes (events), then it is likely that it is going to benefit from applying an <a href="https://en.wikipedia.org/wiki/Event-driven_architecture">Event-drived architecture</a>. But if it is a regular RPC (REST/SOAP/etc), with relatively large traffic between peers (say up to couple of megabytes JSON sent from a server as a response to a client), then using Actor Model is going to be useless at best and maybe even harmful at worst.</p>
<p>I’ve seen a lot of troubles where Actor Model was misused, but probably the most harmful one is to block the thread while processing a message in a context of an Actor. All the Actors in the Actor System are likely being run on some kind of thread pool, thus blocking Actor thread pretty much partially blocks the capacity of the whole Actor System to make progress. Reason for such misuse is likely a lack of understanding how specific actor framework works under the hood, as well as lack of knowledge how to operate with async primitives (e.g. <code class="language-plaintext highlighter-rouge">Future</code>s in an actor context). You don’t need actors to implement request-response pattern! Just use any web-framework FFS! Check out <a href="https://doc.akka.io/docs/akka/current/typed/interaction-patterns.html">Interaction Patterns</a> for more details.</p>
<p>Note on type safety. When dealing with Actor Model, a useful abstraction is so called “Location Transparency”, meaning that the sender of the message does not know where the received would be (meaning the same host or another host, acessible over the network). So in order to send/receive messages over the network in a generic way, the only type available is <code class="language-plaintext highlighter-rouge">&[u8]</code> (or owned <code class="language-plaintext highlighter-rouge">Vec<u8></code>). So the Actor Model is absolutely type-safe, if you know what I mean.</p>
<p>When desigining my own implementation of the Actor Model, I used a metaphore of a post office, and it worked out pretty nice. The Actors can communicate across the network (effectively implementing “Location Transparency”), and implement interesting protocols like <a href="https://github.com/sergey-melnychuk/uppercut/blob/develop/examples/paxos.rs">PAXOS consensus</a> and <a href="https://github.com/sergey-melnychuk/uppercut/blob/develop/examples/gossip.rs">Gossip membership</a>.</p>
<p>Learn more about Uppercut:</p>
<ul>
<li><a href="https://github.com/sergey-melnychuk/uppercut/blob/develop/doc/design-guide.md">Uppercut Design Guide</a></li>
<li><a href="https://github.com/sergey-melnychuk/uppercut-lab/tree/master/uppercut-mio-server">Uppercut Mio Server</a></li>
</ul>Sergey MelnychukTL/DR: read [1] and watch [2]: Actor Model of Computation The Actor Model (Hewitt, Meijer, Szyperski)Cloudflare Workers setup and usage2023-05-19T13:11:42+00:002023-05-19T13:11:42+00:00https://sergey-melnychuk.github.io/2023/05/19/cloudflare-workers<p>TL;DR: See <a href="https://github.com/sergey-melnychuk/project-one/">sergey-melnychuk/project-one</a>.</p>
<p>At the time of writing this post <a href="https://developers.cloudflare.com/workers/">Cloudflare Workers</a> says:</p>
<blockquote>
<p>Deploy serverless code instantly across the globe to give it exceptional performance, reliability, and scale.</p>
</blockquote>
<p>Basically looks like an AWS Lambda but with benfits of WASM runtime (scalability, performance, no “cold starts”, no network overhead).</p>
<h2 id="setup">Setup</h2>
<p>Install <a href="https://developers.cloudflare.com/workers/wrangler/install-and-update/">wrangler</a> and follow <a href="https://github.com/sergey-melnychuk/project-one/">Project One</a> (note that dependency versions are likely to be outdated):</p>
<h4 id="cargotoml">(Cargo.toml)</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[package]
name = "project-one"
version = "0.1.0"
edition = "2021"
[lib]
crate-type = ["cdylib", "rlib"]
[dependencies]
wasm-bindgen = "0.2"
worker = "0.0.11"
</code></pre></div></div>
<h4 id="wranglertoml">(wrangler.toml)</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>name = "project-one"
compatibility_date = "2022-10-10"
main = "build/worker/shim.mjs"
[build]
command = "worker-build --release"
</code></pre></div></div>
<h4 id="srclibrs">(src/lib.rs)</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>use worker::*;
#[event(fetch)]
pub async fn main(_req: Request, _env: Env, _ctx: Context) -> Result<Response> {
// TODO: do your thing
Response::empty()
}
</code></pre></div></div>
<h4 id="build">Build</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cargo install worker-build
worker-build --release
</code></pre></div></div>
<h4 id="run-locally-development-mode">Run locally (development mode)</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ wrangler dev
⛅️ wrangler 3.0.0
------------------
wrangler dev now uses local mode by default, powered by 🔥 Miniflare and 👷 workerd.
To run an edge preview session for your Worker, use wrangler dev --remote
Running custom build: worker-build --release
[INFO]: 🎯 Checking for the Wasm target...
[INFO]: 🌀 Compiling to Wasm...
Finished release [optimized] target(s) in 0.06s
[INFO]: ⬇️ Installing wasm-bindgen...
[INFO]: Optimizing wasm binaries with `wasm-opt`...
[INFO]: Optional fields missing from Cargo.toml: 'description', 'repository', and 'license'. These are not necessary, but recommended
[INFO]: ✨ Done in 2.43s
[INFO]: 📦 Your wasm pkg is ready to publish at /home/work/Temp/project-one/build.
shim.mjs 11.8kb
⚡ Done in 5ms
⎔ Starting local server...
[mf:inf] Ready on http://127.0.0.1:8787/
<snip>
</code></pre></div></div>
<p>NOTE: Running on Ubuntu 22.04 needs <code class="language-plaintext highlighter-rouge">sudo apt install libc++1</code>. More details <a href="https://github.com/cloudflare/workers-sdk/issues/3262">here</a>.</p>
<h4 id="deploy-to-cloudflare">Deploy to Cloudflare</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># export CLOUDFLARE_API_TOKEN=<snip>
# export CLOUDFLARE_ACCOUNT_ID=<snip>
wrangler publish
</code></pre></div></div>
<p>After publishing, you get a public URL to trigger the worker.</p>Sergey MelnychukTL;DR: See sergey-melnychuk/project-one. INFO: Optional fields missing from Cargo.toml: ‘description’, ‘repository’, and ‘license’. These are not necessary, but recommended INFO: ✨ Done in 2.43s INFO: 📦 Your wasm pkg is ready to publish at /home/work/Temp/project-one/build.AWS Lambda memory usage2023-05-12T13:12:42+00:002023-05-12T13:12:42+00:00https://sergey-melnychuk.github.io/2023/05/12/aws-lambda-simple-memory-usage<p>AWS Lambda shows memory peak-usage, but when memory usage hits the limit, breaking down where exactly too much memory is being used might be a very useful things. The runtime is kind of a black box (at least in my understanding), so any telemetry/measurements need to be baked into the function. For this example, I only describe an approach useful for Rust runtime.</p>
<h2 id="setup">Setup</h2>
<p>Just use <a href="https://www.cargo-lambda.info/guide/getting-started.html">cargo lambda</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>## build from source and install using cargo
cargo install --locked cargo-lambda
## create project
cargo lambda new new-lambda-project && cd new-lambda-project
</code></pre></div></div>
<h4 id="new-lambda-projectcargotoml">new-lambda-project/Cargo.toml</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[package]
name = "new-lambda-project"
version = "0.1.0"
edition = "2021"
# Starting in Rust 1.62 you can use `cargo add` to add dependencies
# to your project.
#
# If you're using an older Rust version,
# download cargo-edit(https://github.com/killercup/cargo-edit#installation)
# to install the `add` subcommand.
#
# Running `cargo add DEPENDENCY_NAME` will
# add the latest version of a dependency to the list,
# and it will keep the alphabetic ordering for you.
[dependencies]
lambda_http = { version = "0.8.0", default-features = false, features = ["apigw_http"] }
lambda_runtime = "0.8.0"
tokio = { version = "1", features = ["macros"] }
tracing = { version = "0.1", features = ["log"] }
tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt"] }
</code></pre></div></div>
<h4 id="new-lambda-projectsrcmainrs">new-lambda-project/src/main.rs</h4>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">use</span> <span class="nn">lambda_http</span><span class="p">::{</span><span class="n">run</span><span class="p">,</span> <span class="n">service_fn</span><span class="p">,</span> <span class="n">Body</span><span class="p">,</span> <span class="n">Error</span><span class="p">,</span> <span class="n">Request</span><span class="p">,</span> <span class="n">RequestExt</span><span class="p">,</span> <span class="n">Response</span><span class="p">};</span>
<span class="c">/// This is the main body for the function.</span>
<span class="c">/// Write your code inside it.</span>
<span class="c">/// There are some code example in the following URLs:</span>
<span class="c">/// - https://github.com/awslabs/aws-lambda-rust-runtime/tree/main/examples</span>
<span class="k">async</span> <span class="k">fn</span> <span class="nf">function_handler</span><span class="p">(</span><span class="n">event</span><span class="p">:</span> <span class="n">Request</span><span class="p">)</span> <span class="k">-></span> <span class="n">Result</span><span class="o"><</span><span class="n">Response</span><span class="o"><</span><span class="n">Body</span><span class="o">></span><span class="p">,</span> <span class="n">Error</span><span class="o">></span> <span class="p">{</span>
<span class="c">// Extract some useful information from the request</span>
<span class="k">let</span> <span class="n">who</span> <span class="o">=</span> <span class="n">event</span>
<span class="nf">.query_string_parameters_ref</span><span class="p">()</span>
<span class="nf">.and_then</span><span class="p">(|</span><span class="n">params</span><span class="p">|</span> <span class="n">params</span><span class="nf">.first</span><span class="p">(</span><span class="s">"name"</span><span class="p">))</span>
<span class="nf">.unwrap_or</span><span class="p">(</span><span class="s">"world"</span><span class="p">);</span>
<span class="k">let</span> <span class="n">message</span> <span class="o">=</span> <span class="nd">format!</span><span class="p">(</span><span class="s">"Hello {who}, this is an AWS Lambda HTTP request"</span><span class="p">);</span>
<span class="c">// Return something that implements IntoResponse.</span>
<span class="c">// It will be serialized to the right response event automatically by the runtime</span>
<span class="k">let</span> <span class="n">resp</span> <span class="o">=</span> <span class="nn">Response</span><span class="p">::</span><span class="nf">builder</span><span class="p">()</span>
<span class="nf">.status</span><span class="p">(</span><span class="mi">200</span><span class="p">)</span>
<span class="nf">.header</span><span class="p">(</span><span class="s">"content-type"</span><span class="p">,</span> <span class="s">"text/html"</span><span class="p">)</span>
<span class="nf">.body</span><span class="p">(</span><span class="n">message</span><span class="nf">.into</span><span class="p">())</span>
<span class="nf">.map_err</span><span class="p">(</span><span class="nn">Box</span><span class="p">::</span><span class="n">new</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="nf">Ok</span><span class="p">(</span><span class="n">resp</span><span class="p">)</span>
<span class="p">}</span>
<span class="nd">#[tokio::main]</span>
<span class="k">async</span> <span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="k">-></span> <span class="n">Result</span><span class="o"><</span><span class="p">(),</span> <span class="n">Error</span><span class="o">></span> <span class="p">{</span>
<span class="nn">tracing_subscriber</span><span class="p">::</span><span class="nf">fmt</span><span class="p">()</span>
<span class="nf">.with_max_level</span><span class="p">(</span><span class="nn">tracing</span><span class="p">::</span><span class="nn">Level</span><span class="p">::</span><span class="n">INFO</span><span class="p">)</span>
<span class="c">// disable printing the name of the module in every log line.</span>
<span class="nf">.with_target</span><span class="p">(</span><span class="k">false</span><span class="p">)</span>
<span class="c">// disabling time is handy because CloudWatch will add the ingestion time.</span>
<span class="nf">.without_time</span><span class="p">()</span>
<span class="nf">.init</span><span class="p">();</span>
</code></pre></div></div>
<h2 id="memory-usage">Memory Usage</h2>
<p>Slightly modified version of an approach using <a href="https://docs.rs/jemalloc-ctl/latest/jemalloc_ctl/index.html">jemalloc</a> described on <a href="https://stackoverflow.com/a/30983834">Stack Overflow</a>.</p>
<p>Add necessary dependencies:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[dependencies]
...
jemallocator = "0.5"
jemalloc-sys = {version = "0.5", features = ["stats"]}
libc = "0.2"
jemalloc-ctl = "0.5.0"
</code></pre></div></div>
<p>Add global allocator definition and use snapshot memory usage before/after memory-intense operations:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#[global_allocator]
static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;
fn memory(tag: &str) {
use jemalloc_ctl::{epoch, stats};
epoch::advance().unwrap();
let active = stats::active::read().unwrap() >> 20 + 1;
let allocated = stats::allocated::read().unwrap() >> 20 + 1;
let resident = stats::resident::read().unwrap() >> 20 + 1;
let metadata = stats::metadata::read().unwrap() >> 20 + 1;
tracing::warn!("MEMORY '{}': active: {} Mb, allocated: {} Mb, resident: {} Mb, metadata: {} Mb",
tag, active, allocated, resident, metadata);
}
async fn do_stuff() -> Result<(), Error> {
memory("before");
// do this
memory("this");
// do that
memory("that");
}
</code></pre></div></div>
<h2 id="next-steps">Next Steps</h2>
<p>With memory stats dumped and memory “hot spots” detected, they can be addressed:</p>
<ul>
<li><a href="https://nnethercote.github.io/perf-book/title-page.html">Rest Perf book</a></li>
<li><a href="https://valgrind.org/docs/manual/dh-manual.html">Valgrind DHAT</a></li>
<li><a href="https://docs.rs/dhat/latest/dhat/">DHAT crate</a></li>
<li><a href="https://rust-analyzer.github.io/blog/2020/12/04/measuring-memory-usage-in-rust.html">“Measuring Memory Usage in Rust”</a></li>
<li><a href="https://blog.mozilla.org/nnethercote/2015/06/03/measuring-data-structure-sizes-firefox-c-vs-servo-rust/">“Measuring data structure sizes”</a></li>
</ul>Sergey MelnychukAWS Lambda shows memory peak-usage, but when memory usage hits the limit, breaking down where exactly too much memory is being used might be a very useful things. The runtime is kind of a black box (at least in my understanding), so any telemetry/measurements need to be baked into the function. For this example, I only describe an approach useful for Rust runtime.The mythical 10x engineer2022-09-15T04:57:42+00:002022-09-15T04:57:42+00:00https://sergey-melnychuk.github.io/2022/09/15/10x-engineer<p>The 10x engineer - whole industry was discussing it, laughing at it, learning how to become it quite for some time. Googling “10x engineer” gives tons of content (as probably with any other catch-phrase), including some stuff to <a href="https://www.youtube.com/watch?v=Iydpa_gPdes">watch</a> and <a href="https://a16z.com/2014/07/30/the-happy-demise-of-the-10x-engineer/">read</a>. Quite a lot whas told & written about it, but I feel the World needs one more post on the topic!</p>
<p>// TL/DR: The 10x engineer just knows which 90% of work <strong>not</strong> to do.</p>
<p>At this point after more than 12 years in the industry, I feel that some patterns emerged and crystalized about how the work is done in various environments by various people. With time, I clearly see that in software engineering (where you can build pretty much everythin you can think of, with some limitations mostly in accounting and physics departments) it is more important to choose what <strong>not</strong> to do rather that what to do. By “doing” I mean actually investing time and effort: designing, writing code, fixing it, building pipelines, setting up environments, and eventually, running it in production environment (or any other environment where it can add some value).</p>
<p>Putting it to extreme: probably the worst thing one can do when solving a software engineering task at work is to start writing code. Of course the task needs to be understood first, then necesasry trade-offs selected, the suggested approach verified and designed, then the design reviewed by other engineers. After that the easy part comes in - writing code. The art of (software) engineering IMO is to see the problem behind the task and understand how exactly it needs to be solved (the purpose of the solution). Without such “homework” done, pretty much all the efforts are wasted (unless it is a prototying/research efforts to start with of course). As if all you have is a hammer, every problem looks like a nail, <a href="https://en.wikipedia.org/wiki/Law_of_the_instrument">right</a>?</p>
<p>It is so obvious I even feel weird writing about it. Yet if it is so obvious to everyone, why there are some many failed software projects around? Even when <a href="https://en.wikipedia.org/wiki/The_Mythical_Man-Month">the answers</a> are available since 1975? (Feel free to put into comments: if you read the book and the result of substraction 1975 from the year you were born). Yet engineering management is totally different from software engineering itself, I still can help but notice the same main goal: choose which things <strong>not</strong> to do. When the deadline is (too) close, the rational decision would be to drop stories out of the backlog, isn’t it? (If it does not seem rational to you - just read the damn book already).</p>
<p>How to define what 10x engineer is about? I think 2 questions about an engineer and their work is enough:</p>
<ol>
<li>Can [an engineer] get the work done alone?
<ul>
<li>Alone meaning without (constant) supervision.</li>
<li>If a person can plan/align their workload efficiently, there might be a good chance such person can organize small (or maybe not so small) teams work as well.</li>
</ul>
</li>
<li>Is the work [done by the engineer] being redone necessary?
<ul>
<li>If yes, then usually by more senior colleagues.</li>
<li>Solution might solve the problem, but might be too complex or have limited scalability.</li>
</ul>
</li>
</ol>
<p>These 2 questions give 4 quadrants, I will use <code class="language-plaintext highlighter-rouge">ALONE</code> (<code class="language-plaintext highlighter-rouge">true</code> if an engineer can work alone) and <code class="language-plaintext highlighter-rouge">REDONE</code> (<code class="language-plaintext highlighter-rouge">true</code> if work needs to be redone) as booleans to distinguish quadrants:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">!ALONE && REDONE</code>: Usually Junior Engineers land here, they need supervision and often a few iterations of polishing is required (which is not necessary a bad thing - it allows sustainable learning and improvement).</li>
<li><code class="language-plaintext highlighter-rouge">!ALONE && !REDONE</code>: Junior+/Middle Engineers who require little-to-no supervision (yet still most likely require at least some) and produce production-ready code without significant issues. I’d say starting from here most of the routine tasks can be offloaded to the tooling (auto-formatters, test-coverage analyzers, statical analyzers, various sanitizers and profilers, and other stuff like that).</li>
<li><code class="language-plaintext highlighter-rouge">ALONE && REDONE</code>: Tricky one, I think this quadrant is the result of something like <a href="https://en.wikipedia.org/wiki/Second-system_effect">“Second System Effect”</a> but at a much smaller scale. I regularly visit this quadrant, to be honest, but with some self-awareness it become quite easy to catch drift in this direction early.</li>
<li><code class="language-plaintext highlighter-rouge">ALONE && !REDONE</code>: It seems rational to hunt the beast we’re looking for in this quadrant. Senior Engineers, consistent performers who get the job done live here. Is it enough to land here to become 10x engineer? Maybe. I don’t know! Noone really knows for sure.</li>
</ul>
<p>So how to distinguish a 10x engineer and a 1x one, both living in <code class="language-plaintext highlighter-rouge">ALONE && !REDONE</code> quadrant? I think that “remaining” 9x is where to look for the answer. I think deep understanding of what is happening and why allows mythical 10x engineer to consistently avoid doing unnecessary 90% of work! Almost like <a href="https://en.wikipedia.org/wiki/Pareto_principle">Pareto principle</a> but with numbers twisted to match 10/90 breakdown instead of 20/80 :) I’ve seen tons of efforts wasted on solving problems that simply do not have to exist at all! Like the ones below (don’t get me wrong, not only <em>someone else</em> does it wrong, some of those are also my mistakes):</p>
<ul>
<li>Choosing non-ACID-compliant database and implementing “manual consistency” on top of it</li>
<li>Making all the code (even strictly CPU-bound) async and then struggling to extend it</li>
<li>Blocking the executor threads of reactive app and then struggle with poor performance</li>
<li>Leaking of the implementation details into the domain model and failing to reuse code</li>
<li>Designing everything to work on single instance only and failing to scale out</li>
<li>Choosing tools that simply are not fit for the job and “tweaking” them constantly</li>
</ul>
<p>Those problems, that should have never existed in the first place, can easily dominate the effort invested in the project and IMO can as easily go to 90% of all work done on the project, as such problems tend to compound, effectively setting up the foundations for <a href="https://en.wikipedia.org/wiki/Diminishing_returns">Diminishing Returns</a>. The mythical 10x engineer can foresee such problems and avoid them. This is how the work on the project gets done 10x faster, compared to projects where 90% of efforts are wasted due to a bloated backlog full of problems that could have been avoided. So to summarize: it seems it is more important what <strong>not</strong> to do, what code <strong>not</strong> to write and what feature <strong>not</strong> to ship.</p>
<p>Is it the perfect definition of a mythical 10x engineer? Probably not. But I think each and everyone involved in software engineering might benefit significantly by using “how mythical 10x engineer would solve that?” and “how can we avoid doing that?” mental models and looking for a ways to make solution more simple rather than more complex. Just like in <a href="https://pl.wikipedia.org/wiki/TRIZ">TRIZ</a>: “The ideal system is the one that does not even exist, yet its function is being performed”.</p>Sergey MelnychukThe 10x engineer - whole industry was discussing it, laughing at it, learning how to become it quite for some time. Googling “10x engineer” gives tons of content (as probably with any other catch-phrase), including some stuff to watch and read. Quite a lot whas told & written about it, but I feel the World needs one more post on the topic!Argumentz: small & simple command-line arguments parser in Java2020-07-31T07:35:42+00:002020-07-31T07:35:42+00:00https://sergey-melnychuk.github.io/2020/07/31/introducing-argumentz<p>Every now and then there is a need to pass arguments to the command-line application. While there are a lot of libraries/frameworks to do that in Java, I still had this feeling “why such an easy thing must be so cumbersome and require so much boilerplate?”.</p>
<p>TL/DR: <a href="https://github.com/sergey-melnychuk/argumentz">code</a>.</p>
<p>So I decided: the world needs one more command-line arguments parser for Java. Why? Well, maybe just because I can! First thing to decide on a new project is a name: ‘argumentz’ was available (according to maven central and one popular web search engine). ‘Z’ in ‘Argumentz’ stands for the last letter in the alphabet: I wanted to make the library as simple as possible (but not simpler) and keep user in the total control, so there cannot be anything simpler beyond it, like there are no letters after Z.</p>
<h3 id="example-application">Example application</h3>
<p>The example application takes following command-line arguments: host (string), port (integer), timeout in seconds (integer), user name (string) and a flag to enable verbose logging. So the execution will look like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-cp</span> <span class="nb">.</span> App <span class="nt">-u</span> admin <span class="nt">-h</span> localhost <span class="nt">-p</span> 8080 <span class="nt">-s</span> 30 <span class="nt">-v</span>
</code></pre></div></div>
<p>Or with full names of flags:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-cp</span> <span class="nb">.</span> App <span class="nt">--user</span> admin <span class="nt">--host</span> localhost <span class="nt">--port</span> 8080 <span class="nt">--seconds</span> 30 <span class="nt">--verbose</span>
</code></pre></div></div>
<p>The default value for <code class="language-plaintext highlighter-rouge">user</code> is “guest”, for <code class="language-plaintext highlighter-rouge">port</code> 8080, so if one of those arguments is missing, application will use default values. Each flag has a default value as false, meaning it is disabled until explicitly enabled, pretty standard behavior for a flag.</p>
<h3 id="setup">Setup</h3>
<p>To start, checking out prepared empty project from <a href="https://github.com/sergey-melnychuk/empty-project">empty-project</a> and slightly cleaning it up:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>git clone https://github.com/sergey-melnychuk/empty-project.git <span class="se">\</span>
<span class="nt">-b</span> maven-java8-junit5 <span class="se">\</span>
args-parser
<span class="nv">$ </span><span class="nb">cd </span>args-parser
<span class="nv">$ </span><span class="nb">rm</span> <span class="nt">-rf</span> .git <span class="c"># remove any association with 'empty-project' repository</span>
</code></pre></div></div>
<p>Adding dependency to <code class="language-plaintext highlighter-rouge">pom.xml</code>:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><dependency></span>
<span class="nt"><groupId></span>io.github.sergey-melnychuk<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>argumentz<span class="nt"></artifactId></span>
<span class="nt"><version></span>0.3.9<span class="nt"></version></span>
<span class="nt"></dependency></span>
</code></pre></div></div>
<h3 id="describe-command-line-arguments">Describe command-line arguments</h3>
<p>To describe desired setup in Arguments, <code class="language-plaintext highlighter-rouge">Arguments.builder()</code> is used:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Argumentz</span> <span class="n">arguments</span> <span class="o">=</span> <span class="nc">Argumentz</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span>
<span class="o">.</span><span class="na">withParam</span><span class="o">(</span><span class="sc">'u'</span><span class="o">,</span> <span class="s">"user"</span><span class="o">,</span> <span class="s">"username to connect to the server"</span><span class="o">,</span> <span class="o">()</span> <span class="o">-></span> <span class="s">"guest"</span><span class="o">)</span>
<span class="o">.</span><span class="na">withParam</span><span class="o">(</span><span class="sc">'p'</span><span class="o">,</span> <span class="s">"port"</span><span class="o">,</span> <span class="s">"port for server to listen"</span><span class="o">,</span> <span class="nl">Integer:</span><span class="o">:</span><span class="n">parseInt</span><span class="o">,</span> <span class="o">()</span> <span class="o">-></span> <span class="mi">8080</span><span class="o">)</span>
<span class="o">.</span><span class="na">withParam</span><span class="o">(</span><span class="sc">'s'</span><span class="o">,</span> <span class="s">"seconds"</span><span class="o">,</span> <span class="s">"timeout in seconds"</span><span class="o">,</span> <span class="nl">Integer:</span><span class="o">:</span><span class="n">parseInt</span><span class="o">)</span>
<span class="o">.</span><span class="na">withParam</span><span class="o">(</span><span class="sc">'h'</span><span class="o">,</span> <span class="s">"host"</span><span class="o">,</span> <span class="s">"host for client to connect to"</span><span class="o">)</span>
<span class="o">.</span><span class="na">withFlag</span><span class="o">(</span><span class="sc">'v'</span><span class="o">,</span> <span class="s">"verbose"</span><span class="o">,</span> <span class="s">"enable extra logging"</span><span class="o">)</span>
<span class="o">.</span><span class="na">withErrorHandler</span><span class="o">((</span><span class="nc">RuntimeException</span> <span class="n">e</span><span class="o">,</span> <span class="nc">Argumentz</span> <span class="n">a</span><span class="o">)</span> <span class="o">-></span> <span class="o">{</span>
<span class="nc">System</span><span class="o">.</span><span class="na">err</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">e</span><span class="o">.</span><span class="na">getMessage</span><span class="o">()</span> <span class="o">+</span> <span class="s">"\n"</span><span class="o">);</span>
<span class="n">a</span><span class="o">.</span><span class="na">printUsage</span><span class="o">(</span><span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">);</span>
<span class="nc">System</span><span class="o">.</span><span class="na">exit</span><span class="o">(</span><span class="mi">1</span><span class="o">);</span>
<span class="o">})</span>
<span class="o">.</span><span class="na">build</span><span class="o">();</span>
</code></pre></div></div>
<p>So each argument is either a parameter (<code class="language-plaintext highlighter-rouge">String</code> or actually any type that can be parsed from it) or a flag (just a parameter of type <code class="language-plaintext highlighter-rouge">boolean</code>). Adding parameter with short form <code class="language-plaintext highlighter-rouge">-p</code> and long one <code class="language-plaintext highlighter-rouge">--param</code> can be done in one of the following ways:</p>
<ul>
<li>Required, of type <code class="language-plaintext highlighter-rouge">String</code>: <code class="language-plaintext highlighter-rouge">.withParam('p', "param", "description goes here")</code></li>
<li>Required, produced from <code class="language-plaintext highlighter-rouge">String</code> (in this case <code class="language-plaintext highlighter-rouge">int</code>): <code class="language-plaintext highlighter-rouge">.withParam('p', "param", "...", Integer::parseInt)</code>
<ul>
<li>Any type <code class="language-plaintext highlighter-rouge">T</code> is supported, as long as instance of <code class="language-plaintext highlighter-rouge">Function<String, T></code> is provided.</li>
</ul>
</li>
<li>Not required <code class="language-plaintext highlighter-rouge">String</code> (thus having default value): <code class="language-plaintext highlighter-rouge">.withParam('p', "param", "...", () -> "default")</code></li>
<li>Not required of any type (e.g. <code class="language-plaintext highlighter-rouge">int</code>): <code class="language-plaintext highlighter-rouge">.withParam('p', "param", "...", Integer::parseInt, () -> 42)</code></li>
</ul>
<p>Flag is defined with just <code class="language-plaintext highlighter-rouge">.withFlag('f', "flag", "...")</code>. Distinguish flags between mandatory or optional does not make much sense to me, as well as providing a default value (it is just <code class="language-plaintext highlighter-rouge">false</code>). So flag, is just, well, a flag - either it is there or not.</p>
<p>Error handler provided with <code class="language-plaintext highlighter-rouge">.withErrorHandler</code> is called when the problem arises (missing required argument or parsing arguments value from <code class="language-plaintext highlighter-rouge">String</code> failed). In this example, the error message is printed to <code class="language-plaintext highlighter-rouge">stderr</code>, usage instructions are printed to <code class="language-plaintext highlighter-rouge">stdout</code> and application terminates. One required thing for such error handler to do is to break the execution flow of an application: either throw a runtime exceptions or call <code class="language-plaintext highlighter-rouge">System.exit</code>. Reasons for this requirement are simple: fail fast and let user keep the control.</p>
<h3 id="match-against-provided-arguments">Match against provided arguments</h3>
<p>After there is a definition of <code class="language-plaintext highlighter-rouge">Argumentz</code>, it is possible to <code class="language-plaintext highlighter-rouge">match</code> arguments from <code class="language-plaintext highlighter-rouge">String args[]</code>:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Argumentz</span><span class="o">.</span><span class="na">Match</span> <span class="n">match</span> <span class="o">=</span> <span class="n">arguments</span><span class="o">.</span><span class="na">match</span><span class="o">(</span><span class="n">args</span><span class="o">);</span>
</code></pre></div></div>
<p>After that, all matched values are available from the <code class="language-plaintext highlighter-rouge">match</code> instance. No binding to class properties via annotattions, just plain and simple <code class="language-plaintext highlighter-rouge">.get{,Int,Flag}(String name)</code>, inspired by <a href="https://github.com/lightbend/config">Lightbend Config</a>. Getting all the actual values is pretty straightforward:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">String</span> <span class="n">user</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"user"</span><span class="o">);</span>
<span class="kt">int</span> <span class="n">port</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="na">getInt</span><span class="o">(</span><span class="s">"port"</span><span class="o">);</span>
<span class="nc">String</span> <span class="n">host</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"host"</span><span class="o">);</span>
<span class="kt">int</span> <span class="n">seconds</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="na">getInt</span><span class="o">(</span><span class="s">"seconds"</span><span class="o">);</span>
<span class="kt">boolean</span> <span class="n">verbose</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="na">getFlag</span><span class="o">(</span><span class="s">"verbose"</span><span class="o">);</span>
</code></pre></div></div>
<p>For generic value of type <code class="language-plaintext highlighter-rouge">T</code>, use <code class="language-plaintext highlighter-rouge">match.getAs(Class<T> clazz, String name)</code>. For example if <code class="language-plaintext highlighter-rouge">T</code> is <code class="language-plaintext highlighter-rouge">Param</code>, the call is <code class="language-plaintext highlighter-rouge">match.getAs(Param.class, "param")</code>. The casting from <code class="language-plaintext highlighter-rouge">Object</code> to <code class="language-plaintext highlighter-rouge">T</code> is done by <code class="language-plaintext highlighter-rouge">Class<T>::cast</code>, so if anything goes wrong, <code class="language-plaintext highlighter-rouge">ClassCastException</code> will land into error handler.</p>
<h3 id="error-handling">Error handling</h3>
<p>If an exception is re-thrown from error handler, it will be thrown from <code class="language-plaintext highlighter-rouge">arguments.match</code>. As promised, the user keeps total control under the situation - how to handle the exception is totally up to the user. For example, with <code class="language-plaintext highlighter-rouge">try-catch</code> around <code class="language-plaintext highlighter-rouge">arguments.match</code> it is easy to extract into pure function all code related to the command-line arguments:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">static</span> <span class="nc">Optional</span><span class="o"><</span><span class="nc">Params</span><span class="o">></span> <span class="nf">fetchParams</span><span class="o">(</span><span class="nc">String</span> <span class="n">args</span><span class="o">[])</span> <span class="o">{</span>
<span class="nc">Argumentz</span> <span class="n">arguments</span> <span class="o">=</span> <span class="nc">Argumentz</span><span class="o">.</span><span class="na">builder</span><span class="o">()</span>
<span class="c1">// ...</span>
<span class="o">.</span><span class="na">withErrorHandler</span><span class="o">((</span><span class="nc">RuntimeException</span> <span class="n">e</span><span class="o">,</span> <span class="nc">Argumentz</span> <span class="n">a</span><span class="o">)</span> <span class="o">-></span> <span class="o">{</span> <span class="k">throw</span> <span class="n">e</span><span class="o">;</span> <span class="o">})</span>
<span class="o">.</span><span class="na">build</span><span class="o">();</span>
<span class="k">try</span> <span class="o">{</span>
<span class="nc">Argumentz</span><span class="o">.</span><span class="na">Match</span> <span class="n">match</span> <span class="o">=</span> <span class="n">arguments</span><span class="o">.</span><span class="na">match</span><span class="o">(</span><span class="n">args</span><span class="o">);</span>
<span class="nc">String</span> <span class="n">user</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"user"</span><span class="o">);</span>
<span class="kt">int</span> <span class="n">port</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="na">getInt</span><span class="o">(</span><span class="s">"port"</span><span class="o">);</span>
<span class="nc">String</span> <span class="n">host</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="s">"host"</span><span class="o">);</span>
<span class="kt">int</span> <span class="n">seconds</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="na">getInt</span><span class="o">(</span><span class="s">"seconds"</span><span class="o">);</span>
<span class="kt">boolean</span> <span class="n">verbose</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="na">getFlag</span><span class="o">(</span><span class="s">"verbose"</span><span class="o">);</span>
<span class="k">return</span> <span class="nc">Optional</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="k">new</span> <span class="nc">Params</span><span class="o">(</span><span class="n">host</span><span class="o">,</span> <span class="n">port</span><span class="o">,</span> <span class="n">user</span><span class="o">,</span> <span class="n">seconds</span><span class="o">,</span> <span class="n">verbose</span><span class="o">));</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">IllegalArgumentException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="n">log</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"Failed to fetch Params."</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
<span class="k">return</span> <span class="nc">Optional</span><span class="o">.</span><span class="na">empty</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<h3 id="summary">Summary</h3>
<p>I was presenting <a href="https://github.com/sergey-melnychuk/argumentz">Argumentz</a>, very small, simple and flexible library for parsing command line arguments in Java. It is pure Java, has zero dependencies and has <300 LoC. In the example app, code that handles command-line arguments is straightforward, easy and leaves user in charge.</p>
<p>If you found a bug or a missing feature, please do let me know in comments at the bottom of the page.</p>
<h3 id="ps-publishing-to-maven-central">P.S. Publishing to Maven Central</h3>
<p>The process is pretty much described <a href="https://central.sonatype.org/pages/ossrh-guide.html">here</a>.</p>
<p>I was trying to follow it, but still there were some issues along the way. Below is the summary of what I did to achieve successful release and promotion to Maven Central.</p>
<h4 id="sonatype-jira">Sonatype JIRA</h4>
<ol>
<li>Create an account</li>
<li>Create a <a href="https://issues.sonatype.org/browse/OSSRH-59588">ticket</a></li>
</ol>
<h4 id="gpg">GPG</h4>
<p>Described <a href="https://central.sonatype.org/pages/working-with-pgp-signatures.html">here</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gpg2 <span class="nt">--version</span>
...
<span class="nv">$ </span>gpg2 <span class="nt">--gen-key</span>
...
<span class="nv">$ </span>gpg2 <span class="nt">--list-keys</span>
...
pub rsa2048 2020-07-29 <span class="o">[</span>SC] <span class="o">[</span>expires: 2022-07-29]
<key-id-goes-here>
uid <span class="o">[</span>ultimate] Sergey Melnychuk <sergey-melnychuk@users.noreply.github.com>
sub rsa2048 2020-07-29 <span class="o">[</span>E] <span class="o">[</span>expires: 2022-07-29]
...
<span class="nv">$ </span>gpg2 <span class="nt">--keyserver</span> hkp://pool.sks-keyservers.net <span class="nt">--send-keys</span> <key-id>
gpg: sending key <key-id> to hkp://pool.sks-keyservers.net
</code></pre></div></div>
<h4 id="pomxml">pom.xml</h4>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
<span class="nt"><groupId></span>io.github.sergey-melnychuk<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>argumentz<span class="nt"></artifactId></span>
<span class="nt"><version></span>0.3.9-SNAPSHOT<span class="nt"></version></span>
<span class="nt"><packaging></span>jar<span class="nt"></packaging></span>
<span class="nt"><name></span>${project.groupId}:${project.artifactId}<span class="nt"></name></span>
<span class="nt"><description></span>Command-line arguments parser in Java. Small, simple and flexible.<span class="nt"></description></span>
<span class="nt"><url></span>https://github.com/sergey-melnychuk/argumentz<span class="nt"></url></span>
<span class="nt"><licenses></span>
<span class="nt"><license></span>
<span class="nt"><name></span>Apache License, Version 2.0<span class="nt"></name></span>
<span class="nt"><url></span>http://www.apache.org/licenses/LICENSE-2.0.txt<span class="nt"></url></span>
<span class="nt"><distribution></span>repo<span class="nt"></distribution></span>
<span class="nt"></license></span>
<span class="nt"></licenses></span>
<span class="nt"><developers></span>
<span class="nt"><developer></span>
<span class="nt"><name></span>Sergey Melnychuk<span class="nt"></name></span>
<span class="nt"><url></span>https://github.com/sergey-melnychuk/<span class="nt"></url></span>
<span class="nt"></developer></span>
<span class="nt"></developers></span>
<span class="nt"><scm></span>
<span class="nt"><connection></span>scm:git:git://github.com/sergey-melnychuk/argumentz.git<span class="nt"></connection></span>
<span class="nt"><developerConnection></span>scm:git:git@github.com:sergey-melnychuk/argumentz.git<span class="nt"></developerConnection></span>
<span class="nt"><url></span>http://github.com/sergey-melnychuk/argumentz/tree/master<span class="nt"></url></span>
<span class="nt"></scm></span>
...
</code></pre></div></div>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
<span class="nt"><distributionManagement></span>
<span class="nt"><snapshotRepository></span>
<span class="nt"><id></span>ossrh<span class="nt"></id></span>
<span class="nt"><url></span>https://oss.sonatype.org/content/repositories/snapshots<span class="nt"></url></span>
<span class="nt"></snapshotRepository></span>
<span class="nt"><repository></span>
<span class="nt"><id></span>ossrh<span class="nt"></id></span>
<span class="nt"><url></span>https://oss.sonatype.org/service/local/staging/deploy/maven2/<span class="nt"></url></span>
<span class="nt"></repository></span>
<span class="nt"></distributionManagement></span>
...
</code></pre></div></div>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
<span class="nt"><profiles></span>
<span class="nt"><profile></span>
<span class="nt"><id></span>release<span class="nt"></id></span>
<span class="nt"><build></span>
<span class="nt"><plugins></span>
<span class="nt"><plugin></span>
<span class="nt"><groupId></span>org.sonatype.plugins<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>nexus-staging-maven-plugin<span class="nt"></artifactId></span>
<span class="nt"><version></span>1.6.7<span class="nt"></version></span>
<span class="nt"><extensions></span>true<span class="nt"></extensions></span>
<span class="nt"><configuration></span>
<span class="nt"><serverId></span>ossrh<span class="nt"></serverId></span>
<span class="nt"><nexusUrl></span>https://oss.sonatype.org/<span class="nt"></nexusUrl></span>
<span class="nt"><autoReleaseAfterClose></span>true<span class="nt"></autoReleaseAfterClose></span>
<span class="nt"></configuration></span>
<span class="nt"></plugin></span>
<span class="nt"><plugin></span>
<span class="nt"><groupId></span>org.apache.maven.plugins<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>maven-source-plugin<span class="nt"></artifactId></span>
<span class="nt"><version></span>2.2.1<span class="nt"></version></span>
<span class="nt"><executions></span>
<span class="nt"><execution></span>
<span class="nt"><id></span>attach-sources<span class="nt"></id></span>
<span class="nt"><goals></span>
<span class="nt"><goal></span>jar-no-fork<span class="nt"></goal></span>
<span class="nt"></goals></span>
<span class="nt"></execution></span>
<span class="nt"></executions></span>
<span class="nt"></plugin></span>
<span class="nt"><plugin></span>
<span class="nt"><groupId></span>org.apache.maven.plugins<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>maven-javadoc-plugin<span class="nt"></artifactId></span>
<span class="nt"><version></span>2.9.1<span class="nt"></version></span>
<span class="nt"><executions></span>
<span class="nt"><execution></span>
<span class="nt"><id></span>attach-javadocs<span class="nt"></id></span>
<span class="nt"><goals></span>
<span class="nt"><goal></span>jar<span class="nt"></goal></span>
<span class="nt"></goals></span>
<span class="nt"></execution></span>
<span class="nt"></executions></span>
<span class="nt"></plugin></span>
<span class="nt"><plugin></span>
<span class="nt"><groupId></span>org.apache.maven.plugins<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>maven-gpg-plugin<span class="nt"></artifactId></span>
<span class="nt"><version></span>1.5<span class="nt"></version></span>
<span class="nt"><executions></span>
<span class="nt"><execution></span>
<span class="nt"><id></span>sign-artifacts<span class="nt"></id></span>
<span class="nt"><phase></span>verify<span class="nt"></phase></span>
<span class="nt"><goals></span>
<span class="nt"><goal></span>sign<span class="nt"></goal></span>
<span class="nt"></goals></span>
<span class="nt"></execution></span>
<span class="nt"></executions></span>
<span class="nt"></plugin></span>
<span class="nt"><plugin></span>
<span class="nt"><groupId></span>org.apache.maven.plugins<span class="nt"></groupId></span>
<span class="nt"><artifactId></span>maven-release-plugin<span class="nt"></artifactId></span>
<span class="nt"><version></span>2.5.3<span class="nt"></version></span>
<span class="nt"><configuration></span>
<span class="nt"><autoVersionSubmodules></span>true<span class="nt"></autoVersionSubmodules></span>
<span class="nt"><useReleaseProfile></span>false<span class="nt"></useReleaseProfile></span>
<span class="nt"><releaseProfiles></span>release<span class="nt"></releaseProfiles></span>
<span class="nt"><goals></span>deploy<span class="nt"></goals></span>
<span class="nt"></configuration></span>
<span class="nt"></plugin></span>
<span class="nt"></plugins></span>
<span class="nt"></build></span>
<span class="nt"></profile></span>
<span class="nt"></profiles></span>
...
</code></pre></div></div>
<h4 id="m2settingsxml">~/.m2/settings.xml</h4>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><settings></span>
<span class="nt"><servers></span>
<span class="nt"><server></span>
<span class="nt"><id></span>ossrh<span class="nt"></id></span>
<span class="nt"><username></span>jira-username<span class="nt"></username></span>
<span class="nt"><password></span>jira-password<span class="nt"></password></span>
<span class="nt"></server></span>
<span class="nt"></servers></span>
<span class="nt"><profiles></span>
<span class="nt"><profile></span>
<span class="nt"><id></span>ossrh<span class="nt"></id></span>
<span class="nt"><activation></span>
<span class="nt"><activeByDefault></span>true<span class="nt"></activeByDefault></span>
<span class="nt"></activation></span>
<span class="nt"><properties></span>
<span class="nt"><gpg.executable></span>gpg2<span class="nt"></gpg.executable></span>
<span class="nt"><gpg.passphrase></span>***<span class="nt"></gpg.passphrase></span>
<span class="nt"></properties></span>
<span class="nt"></profile></span>
<span class="nt"></profiles></span>
<span class="nt"></settings></span>
</code></pre></div></div>
<h4 id="release">Release</h4>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mvn release:clean release:prepare <span class="nt">-P</span> release
<span class="nv">$ </span>mvn release:perform <span class="nt">-P</span> release
</code></pre></div></div>
<h4 id="artifacts">Artifacts</h4>
<ul>
<li><a href="https://oss.sonatype.org/service/local/repositories/releases/content/io/github/sergey-melnychuk/argumentz/0.3.9/">Sonatype</a></li>
<li><a href="https://search.maven.org/artifact/io.github.sergey-melnychuk/argumentz/0.3.9/jar">Central</a></li>
</ul>Sergey MelnychukEvery now and then there is a need to pass arguments to the command-line application. While there are a lot of libraries/frameworks to do that in Java, I still had this feeling “why such an easy thing must be so cumbersome and require so much boilerplate?”.Multi-threaded HTTP/WebSocket server in Rust2020-04-27T19:15:42+00:002020-04-27T19:15:42+00:00https://sergey-melnychuk.github.io/2020/04/27/multi-threaded-http-websocket-server-in-rust<p>Building up on my previous posts about <a href="https://sergey-melnychuk.github.io/2019/08/01/rust-mio-tcp-server/">MIO-based server</a> and
<a href="https://sergey-melnychuk.github.io/2019/08/31/rust-parser-combinators/">parser combinators</a>, this post is about making a very simple HTTP
server capable of running on multiple threads and implementing WebSocket protocol.</p>
<p>TL;DR: <a href="https://github.com/sergey-melnychuk/mio-websocket-server">code</a>.</p>
<h3 id="benchmark">Benchmark</h3>
<p>Quick benchmark with <code class="language-plaintext highlighter-rouge">wrk</code> on <code class="language-plaintext highlighter-rouge">8 vCPUs, 30 GB</code> machine shows 110k rps vs 280 rps
when distributing socket reading/writing over 8 threads. <strong>Important note</strong>: this
benchmark is not representative on its own, just the comparison of two allows to
notice 2.5x speedup. <a href="https://en.wikipedia.org/wiki/Amdahl%27s_law">Amdahl’s Law</a>
in action: the main thread is still responsible for listening for incoming
connections and registering socket events.</p>
<h4 id="single-threaded">Single-threaded</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>instance-1:~/mio-tcp-server$ wrk -d 1m -c 128 -t 8 http://127.0.0.1:8080/
Running 1m test @ http://127.0.0.1:8080/
8 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.15ms 131.93us 2.65ms 68.80%
Req/Sec 13.91k 0.86k 19.76k 66.96%
6645523 requests in 1.00m, 557.71MB read
Requests/sec: 110731.94
Transfer/sec: 9.29MB
</code></pre></div></div>
<h4 id="multi-threaded-8-cores-8-threads">Multi-threaded (8 cores, 8 threads)</h4>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>instance-1:~/mio-websocket-server$ wrk -d 1m -c 128 -t 8 http://127.0.0.1:9000/
Running 1m test @ http://127.0.0.1:9000/
8 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 479.65us 554.02us 28.64ms 96.33%
Req/Sec 35.09k 2.01k 59.50k 72.71%
16765024 requests in 1.00m, 1.37GB read
Requests/sec: 279225.41
Transfer/sec: 23.43MB
</code></pre></div></div>
<h3 id="key-concepts">Key Concepts</h3>
<h4 id="handler">Handler</h4>
<p>A container for token, actual socket and send/receive buffers. A <code class="language-plaintext highlighter-rouge">ByteStream</code> from
<a href="https://sergey-melnychuk.github.io/2019/08/31/rust-parser-combinators/">parser-combinators</a>
is useful for stateless parsing (receive stream) and buffered sending (send stream).</p>
<p>Handle wraps a socket provided from listener as a connection, and has <code class="language-plaintext highlighter-rouge">pull()</code> to read from
socket into receive stream, <code class="language-plaintext highlighter-rouge">push()</code> to write data from send stream to the socket, and <code class="language-plaintext highlighter-rouge">put()</code>
to store data for buffering into the send stream.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">Handler</span> <span class="p">{</span>
<span class="n">token</span><span class="p">:</span> <span class="n">Token</span><span class="p">,</span>
<span class="n">socket</span><span class="p">:</span> <span class="n">TcpStream</span><span class="p">,</span>
<span class="n">is_open</span><span class="p">:</span> <span class="nb">bool</span><span class="p">,</span>
<span class="n">recv_stream</span><span class="p">:</span> <span class="n">ByteStream</span><span class="p">,</span>
<span class="n">send_stream</span><span class="p">:</span> <span class="n">ByteStream</span><span class="p">,</span>
<span class="p">}</span>
</code></pre></div></div>
<h4 id="worker-thread">Worker Thread</h4>
<p>Worker Thread receives events (handlers) from the main thread and runs the “payload”.
Then returns handler (most likely in updated state) back to the main thread. All IO events
on connected socket are actually happening on worker threads.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">loop</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">handler</span> <span class="o">=</span> <span class="n">event_rx</span><span class="nf">.lock</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span><span class="nf">.recv</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="nd">debug!</span><span class="p">(</span><span class="s">"token {} background thread"</span><span class="p">,</span> <span class="n">handler</span><span class="py">.token</span><span class="na">.0</span><span class="p">);</span>
<span class="n">handler</span><span class="nf">.pull</span><span class="p">();</span>
<span class="c">// do something useful here</span>
<span class="n">handler</span><span class="nf">.push</span><span class="p">();</span>
<span class="n">ready_tx</span><span class="nf">.send</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The payload (“something useful” part) might be actually parsing the receive buffer:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">handle</span><span class="p">(</span><span class="n">req</span><span class="p">:</span> <span class="n">Request</span><span class="p">)</span> <span class="k">-></span> <span class="n">Response</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span>
<span class="k">if</span> <span class="k">let</span> <span class="nf">Some</span><span class="p">(</span><span class="n">req</span><span class="p">)</span> <span class="o">=</span> <span class="nf">parse_http_request</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">handler</span><span class="py">.recv_stream</span><span class="p">)</span> <span class="p">{</span>
<span class="n">handler</span><span class="py">.recv_stream</span><span class="nf">.pull</span><span class="p">();</span> <span class="c">// roll over the receive stream</span>
<span class="k">let</span> <span class="n">res</span> <span class="o">=</span> <span class="nf">handle</span><span class="p">(</span><span class="n">req</span><span class="p">);</span> <span class="c">// handle the request - get response</span>
<span class="n">handler</span><span class="nf">.put</span><span class="p">(</span><span class="n">res</span><span class="p">);</span> <span class="c">// put response into send stream</span>
<span class="p">}</span>
</code></pre></div></div>
<h4 id="listener-thread">Listener Thread</h4>
<p>The Main Thread owns server socket that receives connections and also <code class="language-plaintext highlighter-rouge">Poll</code> instance, that
allows getting and processing socket events. Once read/write event for specific handler was
received, it is time to send the handler to worker thread for processing.</p>
<p>Meanwhile, handlers that are returning from worker threads need re-registering for next
socket events. So next thing to do for a Listener Thread is to re-register handlers for
respective socket events: if a handler has non-empty send stream, it needs to receive writable
event; and if not the assumption is that it is ready to read some more data.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">loop</span> <span class="p">{</span>
<span class="n">poll</span><span class="nf">.poll</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">events</span><span class="p">,</span> <span class="nf">Some</span><span class="p">(</span><span class="nn">Duration</span><span class="p">::</span><span class="nf">from_millis</span><span class="p">(</span><span class="mi">20</span><span class="p">)))</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="c">// 1. process socket events</span>
<span class="k">for</span> <span class="n">event</span> <span class="n">in</span> <span class="o">&</span><span class="n">events</span> <span class="p">{</span>
<span class="k">match</span> <span class="n">event</span><span class="nf">.token</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Token</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">match</span> <span class="n">listener</span><span class="nf">.accept</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Ok</span><span class="p">((</span><span class="n">socket</span><span class="p">,</span> <span class="mi">_</span><span class="p">))</span> <span class="k">=></span> <span class="p">{</span>
<span class="c">// accept connection, create Handler</span>
<span class="p">},</span>
<span class="nf">Err</span><span class="p">(</span><span class="mi">_</span><span class="p">)</span> <span class="k">=></span> <span class="k">break</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="n">token</span> <span class="k">if</span> <span class="n">event</span><span class="nf">.readiness</span><span class="p">()</span><span class="nf">.is_readable</span><span class="p">()</span> <span class="k">=></span> <span class="p">{</span>
<span class="nd">debug!</span><span class="p">(</span><span class="s">"token {} readable"</span><span class="p">,</span> <span class="n">token</span><span class="na">.0</span><span class="p">);</span>
<span class="k">if</span> <span class="k">let</span> <span class="nf">Some</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span> <span class="o">=</span> <span class="n">handlers</span><span class="nf">.remove</span><span class="p">(</span><span class="o">&</span><span class="n">token</span><span class="p">)</span> <span class="p">{</span>
<span class="n">event_tx</span><span class="nf">.send</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="n">token</span> <span class="k">if</span> <span class="n">event</span><span class="nf">.readiness</span><span class="p">()</span><span class="nf">.is_writable</span><span class="p">()</span> <span class="k">=></span> <span class="p">{</span>
<span class="nd">debug!</span><span class="p">(</span><span class="s">"token {} writable"</span><span class="p">,</span> <span class="n">token</span><span class="na">.0</span><span class="p">);</span>
<span class="k">if</span> <span class="k">let</span> <span class="nf">Some</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span> <span class="o">=</span> <span class="n">handlers</span><span class="nf">.remove</span><span class="p">(</span><span class="o">&</span><span class="n">token</span><span class="p">)</span> <span class="p">{</span>
<span class="n">event_tx</span><span class="nf">.send</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="mi">_</span> <span class="k">=></span> <span class="nd">unreachable!</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c">// 2. process updates received from handlers</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">opt</span> <span class="o">=</span> <span class="n">ready_rx</span><span class="nf">.try_recv</span><span class="p">();</span>
<span class="k">match</span> <span class="n">opt</span> <span class="p">{</span>
<span class="nf">Ok</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span> <span class="k">if</span> <span class="o">!</span><span class="n">handler</span><span class="py">.is_open</span> <span class="k">=></span> <span class="p">{</span>
<span class="c">// socket is closed, drop the handler</span>
<span class="p">},</span>
<span class="nf">Ok</span><span class="p">(</span><span class="n">handler</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">handler</span><span class="py">.send_stream</span><span class="nf">.len</span><span class="p">()</span> <span class="o">></span> <span class="mi">0</span> <span class="p">{</span>
<span class="c">// register handler for writing</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c">// register handler for reading</span>
<span class="p">}</span>
<span class="n">handlers</span><span class="nf">.insert</span><span class="p">(</span><span class="n">handler</span><span class="py">.token</span><span class="p">,</span> <span class="n">handler</span><span class="p">);</span>
<span class="p">},</span>
<span class="mi">_</span> <span class="k">=></span> <span class="k">break</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h4 id="http-to-websocket">HTTP to WebSocket</h4>
<p>WebSocket Upgrade request is just a regular HTTP request, but it needs some special processing, like
calculating ‘Sec-Websocket-Accept’ response header based on ‘Sec-Websocket-Key’ request header below:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">res_sec_websocket_accept</span><span class="p">(</span><span class="n">req_sec_websocket_key</span><span class="p">:</span> <span class="o">&</span><span class="nb">String</span><span class="p">)</span> <span class="k">-></span> <span class="nb">String</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">hasher</span> <span class="o">=</span> <span class="nn">Sha1</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
<span class="n">hasher</span><span class="nf">.input</span><span class="p">(</span><span class="n">req_sec_websocket_key</span><span class="nf">.to_owned</span><span class="p">()</span> <span class="o">+</span> <span class="s">"258EAFA5-E914-47DA-95CA-C5AB0DC85B11"</span><span class="p">);</span>
<span class="nn">base64</span><span class="p">::</span><span class="nf">encode</span><span class="p">(</span><span class="n">hasher</span><span class="nf">.result</span><span class="p">())</span>
<span class="p">}</span>
</code></pre></div></div>
<p>For more details on WebSocket: see nice guide on <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers">MDN</a>
and <a href="https://github.com/sergey-melnychuk/rust-parser-combinators/blob/master/src/ws.rs">rust-parser-combinators</a>. I must admit
parsing binary WebSocket frames is as straightforward as parsing text-based HTTP requests, so parser-combinators seem to do well.</p>
<h4 id="it-works">It Works!</h4>
<p>Finally, putting all pieces together allows connecting to the server from a browser:</p>
<p><img src="https://sergey-melnychuk.github.io/assets/2020-04-27-multi-threaded-http-websocket-server-in-rust/websocket.png" alt="WebSocket connection in Browser" /></p>
<h4 id="plans">Plans</h4>
<ol>
<li>The more I’m moving towards fundamental constructs like sockets and threads, the more code around it seems like Actor Model.
So I have already been <a href="https://github.com/sergey-melnychuk/doing-some-actors">doing-some-actors</a> for some time.</li>
<li>With clean and simple Actor Model implementation and HTTP/WebSocket protocol parser, the classic demo
would be to build… a chat application! This is what is coming next, most likely.</li>
<li>Actor from the Actor Model seems to be way too low-level for direct usage in application-level code.
<ul>
<li>Somehow many people don’t feel wrong writing <code class="language-plaintext highlighter-rouge">class User extend Actor</code> (thus coupling domain-model entity with
specific Actor Model implementation) - for me it seems the same as writing <code class="language-plaintext highlighter-rouge">class User extends Mutex</code>. Just my opinion.</li>
<li>Thus nice and clean (and preferably type-safe) API on top of that might be extremely useful!
Something similar to <a href="https://doc.akka.io/docs/akka/current/stream/stream-quickstart.html">Akka Streams</a> maybe.</li>
</ul>
</li>
</ol>Sergey MelnychukBuilding up on my previous posts about MIO-based server and parser combinators, this post is about making a very simple HTTP server capable of running on multiple threads and implementing WebSocket protocol.Throttling dispatcher in Go2019-09-30T18:27:42+00:002019-09-30T18:27:42+00:00https://sergey-melnychuk.github.io/2019/09/30/throttling-dispatcher-in-go<p>One evening I was thinking why don’t I implement throttling dispatcher in Go. I even had to find <a href="https://www.gopl.io/">“The Go Programming Language”</a> book on the shelf, it was waiting for this for more than a year.</p>
<p>While Go has a lot of controversy in it (like, <a href="https://blog.golang.org/why-generics">no generics</a>, at least for now, and weird/missing dependency management), I must admit it is very simple and very powerful language with “batteries included”: advanced concurrency primitives like goroutines and channels are built into the language, as well as a lot of useful utilities.</p>
<p>Feels like it was created to quickly produce unsophisticated code - calling/service API (e.g. gRPC), intergations, utilities, etc. But even quite meaningful throttling dispatcher that I’ll cover here takes just under 70 lines of code.</p>
<p>The dispatcher is responsible for (eventually) running the submitted task. Throttling in this case is just limiting parallelism - number of simultaneously running tasks.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="n">Scheduler</span> <span class="k">interface</span> <span class="p">{</span>
<span class="n">submit</span><span class="p">(</span><span class="n">f</span> <span class="k">func</span><span class="p">())</span>
<span class="n">stop</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Non-throttling dispatcher is essentially just a (bounded) queue of tasks waiting to be executed:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="n">NonThrottlingScheduler</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">tasks</span> <span class="k">chan</span> <span class="k">func</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">func</span> <span class="n">NewScheduler</span><span class="p">()</span> <span class="o">*</span><span class="n">NonThrottlingScheduler</span> <span class="p">{</span>
<span class="n">tasks</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="k">chan</span> <span class="k">func</span><span class="p">(),</span> <span class="m">1024</span><span class="p">)</span>
<span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">{</span>
<span class="n">task</span><span class="p">,</span> <span class="n">ok</span> <span class="o">:=</span> <span class="o"><-</span><span class="n">tasks</span>
<span class="k">if</span> <span class="o">!</span><span class="n">ok</span> <span class="p">{</span>
<span class="k">break</span>
<span class="p">}</span>
<span class="k">go</span> <span class="n">task</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}()</span>
<span class="k">return</span> <span class="o">&</span><span class="n">NonThrottlingScheduler</span><span class="p">{</span>
<span class="n">tasks</span><span class="o">:</span> <span class="n">tasks</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">func</span> <span class="p">(</span><span class="n">s</span> <span class="o">*</span><span class="n">NonThrottlingScheduler</span><span class="p">)</span> <span class="n">submit</span><span class="p">(</span><span class="n">f</span> <span class="k">func</span><span class="p">())</span> <span class="p">{</span>
<span class="n">s</span><span class="o">.</span><span class="n">tasks</span> <span class="o"><-</span> <span class="n">f</span>
<span class="p">}</span>
<span class="k">func</span> <span class="p">(</span><span class="n">s</span> <span class="o">*</span><span class="n">NonThrottlingScheduler</span><span class="p">)</span> <span class="n">stop</span><span class="p">()</span> <span class="p">{</span>
<span class="nb">close</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">tasks</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Nice thing to have in the scheduler is an opportunity to wait until all tasks have completed their execution. Without it, following program introduces race condition and can finish before any of the tasks get actually completed:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{</span>
<span class="n">scheduler</span> <span class="o">:=</span> <span class="n">NewScheduler</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span> <span class="o">:=</span> <span class="m">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="m">10</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span> <span class="p">{</span>
<span class="n">id</span> <span class="o">:=</span> <span class="n">i</span> <span class="c">// capture the index</span>
<span class="n">scheduler</span><span class="o">.</span><span class="n">submit</span><span class="p">(</span><span class="k">func</span> <span class="p">()</span> <span class="p">{</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%v working...</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">id</span><span class="p">)</span>
<span class="p">})</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"submitted %v</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">id</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">scheduler</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"completed</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>go run main.go
submitted 0
submitted 1
submitted 2
submitted 3
submitted 4
submitted 5
submitted 6
submitted 7
submitted 8
submitted 9
completed
</code></pre></div></div>
<p>To avoid race condition, I’m introducing a new channel, that will only get the object put there when all tasks have finished. Then waiting on this channel will allow waiting for all tasks to complete. Closing the tasks channel before waiting for completing might be a good idea!</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="n">Scheduler</span> <span class="k">interface</span> <span class="p">{</span>
<span class="n">submit</span><span class="p">(</span><span class="n">f</span> <span class="k">func</span><span class="p">())</span>
<span class="n">stop</span><span class="p">()</span>
<span class="n">await</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">type</span> <span class="n">NonThrottlingScheduler</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">tasks</span> <span class="k">chan</span> <span class="k">func</span><span class="p">()</span>
<span class="n">done</span> <span class="k">chan</span> <span class="k">struct</span><span class="p">{}</span>
<span class="p">}</span>
<span class="k">func</span> <span class="n">NewScheduler</span><span class="p">()</span> <span class="o">*</span><span class="n">NonThrottlingScheduler</span> <span class="p">{</span>
<span class="k">var</span> <span class="n">wg</span> <span class="n">sync</span><span class="o">.</span><span class="n">WaitGroup</span>
<span class="n">done</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="k">chan</span> <span class="k">struct</span><span class="p">{})</span>
<span class="n">tasks</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="k">chan</span> <span class="k">func</span><span class="p">(),</span> <span class="m">1024</span><span class="p">)</span>
<span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">{</span>
<span class="n">task</span><span class="p">,</span> <span class="n">ok</span> <span class="o">:=</span> <span class="o"><-</span><span class="n">tasks</span>
<span class="k">if</span> <span class="o">!</span><span class="n">ok</span> <span class="p">{</span>
<span class="k">break</span>
<span class="p">}</span>
<span class="n">wg</span><span class="o">.</span><span class="n">Add</span><span class="p">(</span><span class="m">1</span><span class="p">)</span> <span class="c">// increment the counter when task is received</span>
<span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span>
<span class="k">defer</span> <span class="n">wg</span><span class="o">.</span><span class="n">Done</span><span class="p">()</span> <span class="c">// decrement when taks is completed</span>
<span class="n">task</span><span class="p">()</span>
<span class="p">}()</span>
<span class="p">}</span>
<span class="n">wg</span><span class="o">.</span><span class="n">Wait</span><span class="p">()</span>
<span class="n">done</span> <span class="o"><-</span> <span class="k">struct</span><span class="p">{}{}</span>
<span class="p">}()</span>
<span class="k">return</span> <span class="o">&</span><span class="n">NonThrottlingScheduler</span><span class="p">{</span>
<span class="n">tasks</span><span class="o">:</span> <span class="n">tasks</span><span class="p">,</span>
<span class="n">done</span><span class="o">:</span> <span class="n">done</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">func</span> <span class="p">(</span><span class="n">s</span> <span class="o">*</span><span class="n">NonThrottlingScheduler</span><span class="p">)</span> <span class="n">await</span><span class="p">()</span> <span class="p">{</span>
<span class="o"><-</span><span class="n">s</span><span class="o">.</span><span class="n">done</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Now program finishes only when all tasks complete:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">$</span> <span class="k">go</span> <span class="n">run</span> <span class="n">main</span><span class="o">.</span><span class="k">go</span>
<span class="n">submitted</span> <span class="m">0</span>
<span class="n">submitted</span> <span class="m">1</span>
<span class="n">submitted</span> <span class="m">2</span>
<span class="n">submitted</span> <span class="m">3</span>
<span class="n">submitted</span> <span class="m">4</span>
<span class="n">submitted</span> <span class="m">5</span>
<span class="n">submitted</span> <span class="m">6</span>
<span class="n">submitted</span> <span class="m">7</span>
<span class="n">submitted</span> <span class="m">8</span>
<span class="m">0</span> <span class="n">working</span><span class="o">...</span>
<span class="n">submitted</span> <span class="m">9</span>
<span class="m">2</span> <span class="n">working</span><span class="o">...</span>
<span class="m">8</span> <span class="n">working</span><span class="o">...</span>
<span class="m">1</span> <span class="n">working</span><span class="o">...</span>
<span class="m">5</span> <span class="n">working</span><span class="o">...</span>
<span class="m">6</span> <span class="n">working</span><span class="o">...</span>
<span class="m">7</span> <span class="n">working</span><span class="o">...</span>
<span class="m">4</span> <span class="n">working</span><span class="o">...</span>
<span class="m">3</span> <span class="n">working</span><span class="o">...</span>
<span class="m">9</span> <span class="n">working</span><span class="o">...</span>
<span class="n">completed</span>
</code></pre></div></div>
<p>As for throttling, there is a separate channel for tokens, initiated when schedule is created. Then a task is only gets running when the token is available. Respectively, when a task is completed, the token must be returned back to the channel.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">type</span> <span class="n">ThrottlingScheduler</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">tokens</span> <span class="k">chan</span> <span class="k">struct</span><span class="p">{}</span>
<span class="n">tasks</span> <span class="k">chan</span> <span class="k">func</span><span class="p">()</span>
<span class="n">done</span> <span class="k">chan</span> <span class="k">struct</span><span class="p">{}</span>
<span class="p">}</span>
<span class="k">func</span> <span class="n">NewThrottlingScheduler</span><span class="p">(</span><span class="n">maxParallelism</span> <span class="kt">int</span><span class="p">,</span> <span class="n">maxQueueLength</span> <span class="kt">int</span><span class="p">)</span> <span class="o">*</span><span class="n">ThrottlingScheduler</span> <span class="p">{</span>
<span class="n">done</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="k">chan</span> <span class="k">struct</span><span class="p">{})</span>
<span class="n">tokens</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="k">chan</span> <span class="k">struct</span><span class="p">{},</span> <span class="n">maxParallelism</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="o">:=</span> <span class="m">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">maxParallelism</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span> <span class="p">{</span>
<span class="n">tokens</span> <span class="o"><-</span> <span class="k">struct</span><span class="p">{}{}</span>
<span class="p">}</span>
<span class="k">var</span> <span class="n">wg</span> <span class="n">sync</span><span class="o">.</span><span class="n">WaitGroup</span>
<span class="n">tasks</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="k">chan</span> <span class="k">func</span><span class="p">(),</span> <span class="n">maxQueueLength</span><span class="p">)</span>
<span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">{</span>
<span class="n">task</span><span class="p">,</span> <span class="n">ok</span> <span class="o">:=</span> <span class="o"><-</span><span class="n">tasks</span>
<span class="k">if</span> <span class="o">!</span><span class="n">ok</span> <span class="p">{</span>
<span class="k">break</span>
<span class="p">}</span>
<span class="n">token</span> <span class="o">:=</span> <span class="o"><-</span><span class="n">tokens</span>
<span class="n">wg</span><span class="o">.</span><span class="n">Add</span><span class="p">(</span><span class="m">1</span><span class="p">)</span>
<span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span>
<span class="k">defer</span> <span class="n">wg</span><span class="o">.</span><span class="n">Done</span><span class="p">()</span>
<span class="k">defer</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span>
<span class="n">tokens</span> <span class="o"><-</span> <span class="n">token</span>
<span class="p">}()</span>
<span class="n">task</span><span class="p">()</span>
<span class="p">}()</span>
<span class="p">}</span>
<span class="n">wg</span><span class="o">.</span><span class="n">Wait</span><span class="p">()</span>
<span class="nb">close</span><span class="p">(</span><span class="n">tokens</span><span class="p">)</span>
<span class="n">done</span> <span class="o"><-</span> <span class="k">struct</span><span class="p">{}{}</span>
<span class="p">}()</span>
<span class="k">return</span> <span class="o">&</span><span class="n">ThrottlingScheduler</span> <span class="p">{</span>
<span class="n">tokens</span><span class="o">:</span> <span class="n">tokens</span><span class="p">,</span>
<span class="n">tasks</span><span class="o">:</span> <span class="n">tasks</span><span class="p">,</span>
<span class="n">done</span><span class="o">:</span> <span class="n">done</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">func</span> <span class="n">run</span><span class="p">(</span><span class="n">s</span> <span class="n">Scheduler</span><span class="p">,</span> <span class="n">numTasks</span> <span class="kt">int</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="n">i</span> <span class="o">:=</span> <span class="m">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">numTasks</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span> <span class="p">{</span>
<span class="n">idx</span> <span class="o">:=</span> <span class="n">i</span>
<span class="n">s</span><span class="o">.</span><span class="n">submit</span><span class="p">(</span><span class="k">func</span> <span class="p">()</span> <span class="p">{</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%v working...</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">idx</span><span class="p">)</span>
<span class="n">time</span><span class="o">.</span><span class="n">Sleep</span><span class="p">(</span><span class="m">1</span> <span class="o">*</span> <span class="n">time</span><span class="o">.</span><span class="n">Second</span><span class="p">)</span>
<span class="p">})</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"submitted %v</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">idx</span><span class="p">)</span>
<span class="n">time</span><span class="o">.</span><span class="n">Sleep</span><span class="p">(</span><span class="m">100</span> <span class="o">*</span> <span class="n">time</span><span class="o">.</span><span class="n">Millisecond</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">s</span><span class="o">.</span><span class="n">stop</span><span class="p">()</span>
<span class="n">s</span><span class="o">.</span><span class="n">await</span><span class="p">()</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"completed</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Full throttling scheduler in under 100 lines of code, readable and understandable. Go is not that bad as it seems. Full code is on <a href="https://github.com/sergey-melnychuk/throttling-dispatcher-in-go">GitHub</a>.</p>Sergey MelnychukOne evening I was thinking why don’t I implement throttling dispatcher in Go. I even had to find “The Go Programming Language” book on the shelf, it was waiting for this for more than a year.Monadic parser combinators in Rust2019-08-31T16:24:42+00:002019-08-31T16:24:42+00:00https://sergey-melnychuk.github.io/2019/08/31/rust-parser-combinators<p>Why don’t I implement a nice monadic parser combinator library in Rust? That’s what my thought was when after implementing low-level mock-HTTP server in <a href="https://github.com/sergey-melnychuk/mio-tcp-server">MIO</a> and actually needed to parse the bytes received by server.</p>
<p>What I wanted is a declative way to define sequence of strings to be matched and/or extracted. Example of a parser for a string representing key-value pair (<code class="language-plaintext highlighter-rouge"><string>: <string></code>) would look like this:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[derive(Default)]</span>
<span class="k">struct</span> <span class="n">KV</span> <span class="p">{</span>
<span class="n">k</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="n">v</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">p</span><span class="p">:</span> <span class="n">Parser</span><span class="o"><</span><span class="n">KV</span><span class="o">></span> <span class="o">=</span> <span class="nn">Parser</span><span class="p">::</span><span class="nf">init</span><span class="p">(</span><span class="nn">KV</span><span class="p">::</span><span class="nf">default</span><span class="p">())</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">until</span><span class="p">(</span><span class="sc">':'</span><span class="p">))</span>
<span class="nf">.save</span><span class="p">(|</span><span class="n">target</span><span class="p">,</span> <span class="n">key</span><span class="p">|</span> <span class="n">target</span><span class="py">.k</span> <span class="o">=</span> <span class="n">key</span><span class="p">)</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">exact</span><span class="p">(</span><span class="s">": "</span><span class="p">))</span>
<span class="nf">.save</span><span class="p">(|</span><span class="n">target</span><span class="p">,</span> <span class="n">val</span><span class="p">|</span> <span class="n">target</span><span class="py">.v</span> <span class="o">=</span> <span class="n">val</span><span class="p">);</span>
<span class="k">let</span> <span class="n">stream</span> <span class="o">=</span> <span class="s">"env: prod"</span><span class="nf">.to_stream</span><span class="p">();</span>
<span class="k">let</span> <span class="n">kv</span><span class="p">:</span> <span class="n">KV</span> <span class="o">=</span> <span class="n">p</span><span class="nf">.parse</span><span class="p">(</span><span class="n">stream</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="c">// returns KV { k: "env", v: "prod" }</span>
</code></pre></div></div>
<p>The re-invented stream abstraction is useful when it is required to pull bytes or chunks of bytes from a buffer, with a possibility to reset the position some bytes are read (this allows re-trying matching if needed).</p>
<p>The implementation has a few key points. The <code class="language-plaintext highlighter-rouge">Matcher</code> type - just a function that can be applied to a <code class="language-plaintext highlighter-rouge">ByteStream</code>. <code class="language-plaintext highlighter-rouge">ByteStream</code> itself is just a wrapper over <code class="language-plaintext highlighter-rouge">Vec<u8></code>, nothing fancy.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">type</span> <span class="n">Matcher</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="o">=</span> <span class="n">dyn</span> <span class="nf">Fn</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">ByteStream</span><span class="p">)</span> <span class="k">-></span> <span class="n">Result</span><span class="o"><</span><span class="n">T</span><span class="p">,</span> <span class="n">MatchError</span><span class="o">></span> <span class="o">+</span> <span class="nv">'static</span><span class="p">;</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">single</code> matcher that matches a single character from input <code class="language-plaintext highlighter-rouge">ByteStream</code> looks like this:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">single</span><span class="p">(</span><span class="n">chr</span><span class="p">:</span> <span class="nb">char</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="nb">char</span><span class="o">>></span> <span class="p">{</span>
<span class="nn">Box</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="k">move</span> <span class="p">|</span><span class="n">bs</span><span class="p">|</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">pos</span> <span class="o">=</span> <span class="n">bs</span><span class="nf">.pos</span><span class="p">();</span>
<span class="n">bs</span><span class="nf">.next</span><span class="p">()</span>
<span class="nf">.map</span><span class="p">(|</span><span class="n">b</span><span class="p">|</span> <span class="n">b</span> <span class="k">as</span> <span class="nb">char</span><span class="p">)</span>
<span class="nf">.filter</span><span class="p">(|</span><span class="n">c</span><span class="p">|</span> <span class="o">*</span><span class="n">c</span> <span class="o">==</span> <span class="n">chr</span><span class="p">)</span>
<span class="nf">.ok_or</span><span class="p">(</span><span class="nn">MatchError</span><span class="p">::</span><span class="nf">unexpected</span><span class="p">(</span>
<span class="n">pos</span><span class="p">,</span>
<span class="nd">format!</span><span class="p">(</span><span class="s">"EOF"</span><span class="p">),</span>
<span class="nd">format!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span> <span class="n">chr</span><span class="p">),</span>
<span class="p">))</span>
<span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">repeat</code> matcher is a more generic one, that allows matching zero or more occurrences of an input matcher:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="n">repeat</span><span class="o"><</span><span class="n">T</span><span class="p">:</span> <span class="nv">'static</span><span class="o">></span><span class="p">(</span><span class="n">this</span><span class="p">:</span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">T</span><span class="o">>></span><span class="p">)</span> <span class="k">-></span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="n">T</span><span class="o">>>></span> <span class="p">{</span>
<span class="nn">Box</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="k">move</span> <span class="p">|</span><span class="n">bs</span><span class="p">|</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">acc</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="o">=</span> <span class="nd">vec!</span><span class="p">[];</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">mark</span> <span class="o">=</span> <span class="n">bs</span><span class="nf">.mark</span><span class="p">();</span>
<span class="k">match</span> <span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">)(</span><span class="n">bs</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Err</span><span class="p">(</span><span class="mi">_</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="n">bs</span><span class="nf">.reset</span><span class="p">(</span><span class="n">mark</span><span class="p">);</span>
<span class="k">return</span> <span class="nf">Ok</span><span class="p">(</span><span class="n">acc</span><span class="p">);</span>
<span class="p">}</span>
<span class="nf">Ok</span><span class="p">(</span><span class="n">t</span><span class="p">)</span> <span class="k">=></span> <span class="n">acc</span><span class="nf">.push</span><span class="p">(</span><span class="n">t</span><span class="p">),</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>
<p>There are more useful matchers: <code class="language-plaintext highlighter-rouge">one</code>, <code class="language-plaintext highlighter-rouge">maybe</code>, <code class="language-plaintext highlighter-rouge">until</code>, <code class="language-plaintext highlighter-rouge">before</code>, <code class="language-plaintext highlighter-rouge">token</code>, <code class="language-plaintext highlighter-rouge">string</code>, <code class="language-plaintext highlighter-rouge">space</code> and <code class="language-plaintext highlighter-rouge">bytes</code>. I believe it is possible to infer underlying functionality from the name of a matcher.</p>
<p>To chain two matchers into a new matcher, that represents sequential match made by the first one and then by the second one, there is a <code class="language-plaintext highlighter-rouge">chain</code> operator:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Given Matcher<T> and Matcher<U>, make a Matcher<(T, U)>.</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="n">chain</span><span class="o"><</span><span class="n">T</span><span class="p">:</span> <span class="nv">'static</span><span class="p">,</span> <span class="n">U</span><span class="p">:</span> <span class="nv">'static</span><span class="o">></span><span class="p">(</span>
<span class="n">this</span><span class="p">:</span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">T</span><span class="o">>></span><span class="p">,</span>
<span class="n">next</span><span class="p">:</span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">U</span><span class="o">>></span><span class="p">,</span>
<span class="p">)</span> <span class="k">-></span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="p">(</span><span class="n">T</span><span class="p">,</span> <span class="n">U</span><span class="p">)</span><span class="o">>></span> <span class="p">{</span>
<span class="nn">Box</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="k">move</span> <span class="p">|</span><span class="n">bs</span><span class="p">|</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">t</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">)(</span><span class="n">bs</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">u</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">next</span><span class="p">)(</span><span class="n">bs</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="nf">Ok</span><span class="p">((</span><span class="n">t</span><span class="p">,</span> <span class="n">u</span><span class="p">))</span>
<span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>
<p>To manipulate the matched content, there is a <code class="language-plaintext highlighter-rouge">map</code> operator:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Just apply 'f' on result of a match</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="n">map</span><span class="o"><</span><span class="n">T</span><span class="p">:</span> <span class="nv">'static</span><span class="p">,</span> <span class="n">U</span><span class="p">:</span> <span class="nv">'static</span><span class="p">,</span> <span class="n">F</span><span class="p">:</span> <span class="nf">Fn</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="k">-></span> <span class="n">U</span> <span class="o">+</span> <span class="nv">'static</span><span class="o">></span><span class="p">(</span>
<span class="n">this</span><span class="p">:</span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">T</span><span class="o">>></span><span class="p">,</span>
<span class="n">f</span><span class="p">:</span> <span class="n">F</span><span class="p">,</span>
<span class="p">)</span> <span class="k">-></span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">U</span><span class="o">>></span> <span class="p">{</span>
<span class="nn">Box</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="k">move</span> <span class="p">|</span><span class="n">bs</span><span class="p">|</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">t</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">)(</span><span class="n">bs</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">u</span> <span class="o">=</span> <span class="nf">f</span><span class="p">(</span><span class="n">t</span><span class="p">);</span>
<span class="nf">Ok</span><span class="p">(</span><span class="n">u</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Other operators, like <code class="language-plaintext highlighter-rouge">apply</code>, <code class="language-plaintext highlighter-rouge">expose</code> and <code class="language-plaintext highlighter-rouge">unit</code> follow similar approach.</p>
<p>In current form it is not too convenient to keep bunch of matchers around, so to avoid it I can just “glue” them together for composability into <code class="language-plaintext highlighter-rouge">Parser<T></code>:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">struct</span> <span class="n">Parser</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span>
<span class="n">f</span><span class="p">:</span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">T</span><span class="o">>></span><span class="p">,</span>
<span class="p">}</span>
<span class="k">impl</span><span class="o"><</span><span class="n">T</span><span class="p">:</span> <span class="nv">'static</span><span class="o">></span> <span class="n">Parser</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">unit</span><span class="p">(</span><span class="n">f</span><span class="p">:</span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">T</span><span class="o">>></span><span class="p">)</span> <span class="k">-></span> <span class="n">Parser</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span>
<span class="n">Parser</span> <span class="p">{</span> <span class="n">f</span> <span class="p">}</span>
<span class="p">}</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="n">init</span><span class="o"><</span><span class="n">F</span><span class="p">:</span> <span class="nf">Fn</span><span class="p">()</span> <span class="k">-></span> <span class="n">T</span> <span class="o">+</span> <span class="nv">'static</span><span class="o">></span><span class="p">(</span><span class="n">f</span><span class="p">:</span> <span class="n">F</span><span class="p">)</span> <span class="k">-></span> <span class="n">Parser</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span>
<span class="nn">Parser</span><span class="p">::</span><span class="nf">unit</span><span class="p">(</span><span class="nf">unit</span><span class="p">(</span><span class="n">f</span><span class="p">))</span>
<span class="p">}</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="n">then</span><span class="o"><</span><span class="n">U</span><span class="p">:</span> <span class="nv">'static</span><span class="o">></span><span class="p">(</span><span class="k">self</span><span class="p">,</span> <span class="n">that</span><span class="p">:</span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">U</span><span class="o">>></span><span class="p">)</span> <span class="k">-></span> <span class="n">Parser</span><span class="o"><</span><span class="p">(</span><span class="n">T</span><span class="p">,</span> <span class="n">U</span><span class="p">)</span><span class="o">></span> <span class="p">{</span>
<span class="nn">Parser</span><span class="p">::</span><span class="nf">unit</span><span class="p">(</span><span class="nf">chain</span><span class="p">(</span><span class="k">self</span><span class="py">.f</span><span class="p">,</span> <span class="n">that</span><span class="p">))</span>
<span class="p">}</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="n">map</span><span class="o"><</span><span class="n">U</span><span class="p">:</span> <span class="nv">'static</span><span class="p">,</span> <span class="n">F</span><span class="p">:</span> <span class="nf">Fn</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="k">-></span> <span class="n">U</span> <span class="o">+</span> <span class="nv">'static</span><span class="o">></span><span class="p">(</span><span class="k">self</span><span class="p">,</span> <span class="n">f</span><span class="p">:</span> <span class="n">F</span><span class="p">)</span> <span class="k">-></span> <span class="n">Parser</span><span class="o"><</span><span class="n">U</span><span class="o">></span> <span class="p">{</span>
<span class="nn">Parser</span><span class="p">::</span><span class="nf">unit</span><span class="p">(</span><span class="nf">map</span><span class="p">(</span><span class="k">self</span><span class="py">.f</span><span class="p">,</span> <span class="n">f</span><span class="p">))</span>
<span class="p">}</span>
<span class="o">...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Same operators, but wrapped in <code class="language-plaintext highlighter-rouge">Parser::unit</code> actually allow convenient chaining.</p>
<p>At some point I’ve discovered that in order to define next matcher in a chain, glimpse of a current state of the accumulator is required (example: match the HTTP request body as exactly number of bytes in <code class="language-plaintext highlighter-rouge">Content-Length</code> header), I would need a way to look “into” the monad. Actually what I was looking for is a <code class="language-plaintext highlighter-rouge">flat_map</code> operator. I decided to name it <code class="language-plaintext highlighter-rouge">then_with</code> - just because I can.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="n">expose</span><span class="o"><</span><span class="n">T</span><span class="p">:</span> <span class="nv">'static</span><span class="p">,</span> <span class="n">U</span><span class="p">:</span> <span class="nv">'static</span><span class="p">,</span> <span class="n">F</span><span class="p">:</span> <span class="nf">Fn</span><span class="p">(</span><span class="o">&</span><span class="n">T</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">U</span><span class="o">>></span> <span class="o">+</span> <span class="nv">'static</span><span class="o">></span><span class="p">(</span>
<span class="n">this</span><span class="p">:</span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">T</span><span class="o">>></span><span class="p">,</span>
<span class="n">f</span><span class="p">:</span> <span class="n">F</span><span class="p">,</span>
<span class="p">)</span> <span class="k">-></span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="p">(</span><span class="n">T</span><span class="p">,</span> <span class="n">U</span><span class="p">)</span><span class="o">>></span> <span class="p">{</span>
<span class="nn">Box</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="k">move</span> <span class="p">|</span><span class="n">bs</span><span class="p">|</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">t</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">)(</span><span class="n">bs</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">g</span> <span class="o">=</span> <span class="nf">f</span><span class="p">(</span><span class="o">&</span><span class="n">t</span><span class="p">);</span>
<span class="k">let</span> <span class="n">u</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">g</span><span class="p">)(</span><span class="n">bs</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="nf">Ok</span><span class="p">((</span><span class="n">t</span><span class="p">,</span> <span class="n">u</span><span class="p">))</span>
<span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">impl</span><span class="o"><</span><span class="n">T</span><span class="p">:</span> <span class="nv">'static</span><span class="o">></span> <span class="n">Parser</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span>
<span class="o">...</span>
<span class="k">fn</span> <span class="n">then_with</span><span class="o"><</span><span class="n">U</span><span class="p">:</span> <span class="nv">'static</span><span class="p">,</span> <span class="n">F</span><span class="p">:</span> <span class="nf">Fn</span><span class="p">(</span><span class="o">&</span><span class="n">T</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Box</span><span class="o"><</span><span class="n">Matcher</span><span class="o"><</span><span class="n">U</span><span class="o">>></span> <span class="o">+</span> <span class="nv">'static</span><span class="o">></span><span class="p">(</span><span class="k">self</span><span class="p">,</span> <span class="n">f</span><span class="p">:</span> <span class="n">F</span><span class="p">)</span> <span class="k">-></span> <span class="n">Parser</span><span class="o"><</span><span class="p">(</span><span class="n">T</span><span class="p">,</span> <span class="n">U</span><span class="p">)</span><span class="o">></span> <span class="p">{</span>
<span class="nn">Parser</span><span class="p">::</span><span class="nf">unit</span><span class="p">(</span><span class="nf">expose</span><span class="p">(</span><span class="k">self</span><span class="py">.f</span><span class="p">,</span> <span class="n">f</span><span class="p">))</span>
<span class="p">}</span>
<span class="o">...</span>
</code></pre></div></div>
<p>For example of usage, here is the implementation of HTTP request parser:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[derive(Debug)]</span>
<span class="k">struct</span> <span class="n">Header</span> <span class="p">{</span>
<span class="n">name</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="n">value</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="p">}</span>
<span class="nd">#[derive(Debug,</span> <span class="nd">Default)]</span>
<span class="k">struct</span> <span class="n">Request</span> <span class="p">{</span>
<span class="n">method</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="n">path</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="n">protocol</span><span class="p">:</span> <span class="nb">String</span><span class="p">,</span>
<span class="n">headers</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="n">Header</span><span class="o">></span><span class="p">,</span>
<span class="n">content</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span><span class="p">,</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">header_parser</span><span class="p">()</span> <span class="k">-></span> <span class="n">Parser</span><span class="o"><</span><span class="n">Header</span><span class="o">></span> <span class="p">{</span>
<span class="nn">Parser</span><span class="p">::</span><span class="nf">init</span><span class="p">(||</span> <span class="nd">vec!</span><span class="p">[])</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">before</span><span class="p">(</span><span class="sc">':'</span><span class="p">))</span>
<span class="nf">.map</span><span class="p">(|(</span><span class="k">mut</span> <span class="n">vec</span><span class="p">,</span> <span class="n">val</span><span class="p">)|</span> <span class="p">{</span>
<span class="n">vec</span><span class="nf">.push</span><span class="p">(</span><span class="nf">as_string</span><span class="p">(</span><span class="n">val</span><span class="p">));</span>
<span class="n">vec</span>
<span class="p">})</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">single</span><span class="p">(</span><span class="sc">':'</span><span class="p">))</span>
<span class="nf">.map</span><span class="p">(|(</span><span class="n">vec</span><span class="p">,</span> <span class="mi">_</span><span class="p">)|</span> <span class="n">vec</span><span class="p">)</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">single</span><span class="p">(</span><span class="sc">' '</span><span class="p">))</span>
<span class="nf">.map</span><span class="p">(|(</span><span class="n">vec</span><span class="p">,</span> <span class="mi">_</span><span class="p">)|</span> <span class="n">vec</span><span class="p">)</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">before</span><span class="p">(</span><span class="sc">'\n'</span><span class="p">))</span>
<span class="nf">.map</span><span class="p">(|(</span><span class="k">mut</span> <span class="n">vec</span><span class="p">,</span> <span class="n">val</span><span class="p">)|</span> <span class="p">{</span>
<span class="n">vec</span><span class="nf">.push</span><span class="p">(</span><span class="nf">as_string</span><span class="p">(</span><span class="n">val</span><span class="p">));</span>
<span class="n">vec</span>
<span class="p">})</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">single</span><span class="p">(</span><span class="sc">'\n'</span><span class="p">))</span>
<span class="nf">.map</span><span class="p">(|(</span><span class="n">vec</span><span class="p">,</span> <span class="mi">_</span><span class="p">)|</span> <span class="n">vec</span><span class="p">)</span>
<span class="nf">.map</span><span class="p">(|</span><span class="n">vec</span><span class="p">|</span> <span class="n">Header</span> <span class="p">{</span>
<span class="n">name</span><span class="p">:</span> <span class="n">vec</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="nf">.to_owned</span><span class="p">(),</span>
<span class="n">value</span><span class="p">:</span> <span class="n">vec</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="nf">.to_owned</span><span class="p">(),</span>
<span class="p">})</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">request_parser</span><span class="p">()</span> <span class="k">-></span> <span class="n">Parser</span><span class="o"><</span><span class="n">Request</span><span class="o">></span> <span class="p">{</span>
<span class="nn">Parser</span><span class="p">::</span><span class="nf">init</span><span class="p">(||</span> <span class="nn">Request</span><span class="p">::</span><span class="nf">default</span><span class="p">())</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">before</span><span class="p">(</span><span class="sc">' '</span><span class="p">))</span>
<span class="nf">.save</span><span class="p">(|</span><span class="n">req</span><span class="p">,</span> <span class="n">bytes</span><span class="p">|</span> <span class="n">req</span><span class="py">.method</span> <span class="o">=</span> <span class="nf">as_string</span><span class="p">(</span><span class="n">bytes</span><span class="p">))</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">single</span><span class="p">(</span><span class="sc">' '</span><span class="p">))</span>
<span class="nf">.skip</span><span class="p">()</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">before</span><span class="p">(</span><span class="sc">' '</span><span class="p">))</span>
<span class="nf">.save</span><span class="p">(|</span><span class="n">req</span><span class="p">,</span> <span class="n">bytes</span><span class="p">|</span> <span class="n">req</span><span class="py">.path</span> <span class="o">=</span> <span class="nf">as_string</span><span class="p">(</span><span class="n">bytes</span><span class="p">))</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">single</span><span class="p">(</span><span class="sc">' '</span><span class="p">))</span>
<span class="nf">.skip</span><span class="p">()</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">before</span><span class="p">(</span><span class="sc">'\n'</span><span class="p">))</span>
<span class="nf">.save</span><span class="p">(|</span><span class="n">req</span><span class="p">,</span> <span class="n">bytes</span><span class="p">|</span> <span class="n">req</span><span class="py">.protocol</span> <span class="o">=</span> <span class="nf">as_string</span><span class="p">(</span><span class="n">bytes</span><span class="p">))</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">single</span><span class="p">(</span><span class="sc">'\n'</span><span class="p">))</span>
<span class="nf">.skip</span><span class="p">()</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">repeat</span><span class="p">(</span><span class="nf">header_parser</span><span class="p">()</span><span class="nf">.into</span><span class="p">()))</span>
<span class="nf">.save</span><span class="p">(|</span><span class="n">req</span><span class="p">,</span> <span class="n">vec</span><span class="p">|</span> <span class="n">req</span><span class="py">.headers</span> <span class="o">=</span> <span class="n">vec</span><span class="p">)</span>
<span class="nf">.then</span><span class="p">(</span><span class="nf">single</span><span class="p">(</span><span class="sc">'\n'</span><span class="p">))</span>
<span class="nf">.skip</span><span class="p">()</span>
<span class="nf">.then_with</span><span class="p">(|</span><span class="n">req</span><span class="p">|</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">len</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="nf">get_header_value</span><span class="p">(</span><span class="n">req</span><span class="p">,</span> <span class="s">"Content-Length"</span><span class="p">)</span>
<span class="nf">.map</span><span class="p">(|</span><span class="n">n</span><span class="p">|</span> <span class="n">n</span><span class="py">.parse</span><span class="p">::</span><span class="o"><</span><span class="nb">usize</span><span class="o">></span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">())</span>
<span class="nf">.unwrap_or_default</span><span class="p">();</span>
<span class="nf">bytes</span><span class="p">(</span><span class="n">len</span><span class="p">)</span>
<span class="p">})</span>
<span class="nf">.save</span><span class="p">(|</span><span class="n">req</span><span class="p">,</span> <span class="n">content</span><span class="p">|</span> <span class="n">req</span><span class="py">.content</span> <span class="o">=</span> <span class="n">content</span><span class="p">)</span>
<span class="p">}</span>
<span class="c">// Checking if it actually works</span>
<span class="nd">#[cfg(test)]</span>
<span class="k">mod</span> <span class="n">tests</span> <span class="p">{</span>
<span class="k">use</span> <span class="nn">super</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="nd">#[test]</span>
<span class="k">fn</span> <span class="nf">http_request</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">text</span> <span class="o">=</span> <span class="s">r#"GET /docs/index.html HTTP/1.1
Host: www.nowhere123.com
Accept: image/gif, image/jpeg, */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Content-Length: 8
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
0123456
"#</span><span class="p">;</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">bs</span> <span class="o">=</span> <span class="n">text</span><span class="nf">.to_string</span><span class="p">()</span><span class="nf">.into_stream</span><span class="p">();</span>
<span class="k">let</span> <span class="n">req</span> <span class="o">=</span> <span class="nf">request_parser</span><span class="p">()</span><span class="nf">.apply</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">bs</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.method</span><span class="p">,</span> <span class="s">"GET"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.path</span><span class="p">,</span> <span class="s">"/docs/index.html"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.protocol</span><span class="p">,</span> <span class="s">"HTTP/1.1"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.content</span><span class="p">,</span> <span class="n">b</span><span class="s">"0123456</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="py">.name</span><span class="p">,</span> <span class="s">"Host"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="py">.value</span><span class="p">,</span> <span class="s">"www.nowhere123.com"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="py">.name</span><span class="p">,</span> <span class="s">"Accept"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="py">.value</span><span class="p">,</span> <span class="s">"image/gif, image/jpeg, */*"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span><span class="py">.name</span><span class="p">,</span> <span class="s">"Accept-Language"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span><span class="py">.value</span><span class="p">,</span> <span class="s">"en-us"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span><span class="py">.name</span><span class="p">,</span> <span class="s">"Accept-Encoding"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span><span class="py">.value</span><span class="p">,</span> <span class="s">"gzip, deflate"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span><span class="py">.name</span><span class="p">,</span> <span class="s">"Content-Length"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span><span class="py">.value</span><span class="p">,</span> <span class="s">"8"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="py">.name</span><span class="p">,</span> <span class="s">"User-Agent"</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">req</span><span class="py">.headers</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span><span class="py">.value</span><span class="p">,</span> <span class="s">"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"</span><span class="p">);</span>
<span class="p">}</span>
<span class="o">...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This is easy way to build clean, simple and composable parser combinators, glue them into monad, and define naive HTTP request parser, in pure Rust, without any dependencies. And this naive request parser event seems to work! At least test passes.</p>
<p>Current implementation is “stateless” in a way that if entity is not fully matched (e.g. not all chunks of a request were received), it returns error and next attempts will need to start over. For this reason, the client code should care about keeping the current read position in a stream and resetting the stream to that position if matching did not succeed.</p>
<p>The “stateful” parser could keep the aggregate and repeat matching from last position in a stream. I believe it would require to plug in the State monad somewhere, and allow returning something like <code class="language-plaintext highlighter-rouge">(State, Either<Error, Parser<T>>)</code> from parser.</p>
<p>The final code with implemented parsers for HTTP request and WebSocket frame are available on <a href="https://github.com/sergey-melnychuk/rust-parser-combinators">github</a>.</p>Sergey MelnychukWhy don’t I implement a nice monadic parser combinator library in Rust? That’s what my thought was when after implementing low-level mock-HTTP server in MIO and actually needed to parse the bytes received by server.Playing with NASM on MacOS2019-08-27T19:54:42+00:002019-08-27T19:54:42+00:00https://sergey-melnychuk.github.io/2019/08/27/nasm-on-mac<p>Once I was wondering, how different the ‘hello world’ in assembler on Linux would be from one on MacOS.</p>
<p>It appears that not by much. Just some extra tricks required to make things work as expected: taking into account different <code class="language-plaintext highlighter-rouge">write</code> and <code class="language-plaintext highlighter-rouge">exit</code> syscall numbers, as well as adding extra parameters (<code class="language-plaintext highlighter-rouge">-macosx_version_min</code> and <code class="language-plaintext highlighter-rouge">-lSystem</code>) to the linker.</p>
<p>The code is pretty straightforward, as rediscovering ‘hello world’ is hard, thus just taking it from <a href="https://stackoverflow.com/a/21130076">here</a>:</p>
<pre><code class="language-asm">SECTION .text
global _main
_main:
mov rax, 0x2000004 ; syscall 4: write (
mov rdi, 1 ; fd,
mov rsi, Msg ; buffer,
mov rdx, Len ; size
syscall ; )
mov rax, 0x2000001 ; syscall 1: exit (
mov rdi, 0 ; retcode
syscall ; )
SECTION .data
Msg db `Hello, world!\n`
Len: equ $-Msg
</code></pre>
<p>Next it needs to be compiled (by <code class="language-plaintext highlighter-rouge">nasm</code> in this case, can be installed with <code class="language-plaintext highlighter-rouge">brew install nasm</code>) and linked:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nasm -fmacho64 -o hello.o hello.asm
$ ld -o hello hello.o -macosx_version_min 10.13 -lSystem
$ ./hello
Hello, world
</code></pre></div></div>
<p>More details on NASM in MacOS <a href="http://sevanspowell.net/posts/learning-nasm-on-macos.html">here</a>.</p>
<p>Setting up <code class="language-plaintext highlighter-rouge">gdb</code> requires some more tricks with code-signing. Those tricks are described <a href="https://stackoverflow.com/a/32727069">here</a> and <a href="https://stackoverflow.com/a/57398040">here</a>. Short dump of commands for reference:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Keychain Access -> Certificate Assistant -> Create Certificate
# gdb-cert, Self Signed Root, Code Signing, Let me override defaults
# Specify a Location for The Certificate: Keychain = System
# On the certificate: Trust section, Code Signing = Always Trust
$ sudo killall taskgated
$ codesign -fs gdb-cert "$(which gdb)"
$ echo "set startup-with-shell off" > ~/.gdbinit
$ codesign --entitlements gdb.xml -fs gdb-cert /usr/local/bin/gdb
$ cat gdb.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.cs.allow-jit</key>
<true/>
<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
<true/>
<key>com.apple.security.cs.allow-dyld-environment-variables</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<key>com.apple.security.cs.disable-executable-page-protection</key>
<true/>
<key>com.apple.security.cs.debugger</key>
<true/>
<key>com.apple.security.get-task-allow</key>
<true/>
</dict>
</plist>
</code></pre></div></div>
<p>After that <code class="language-plaintext highlighter-rouge">gdb</code> can be run without <code class="language-plaintext highlighter-rouge">sudo</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ gdb hello
GNU gdb (GDB) 8.3
...
Reading symbols from hello...
(No debugging symbols found in hello)
(gdb) b main
Breakpoint 1 at 0x1fd9
(gdb) r
Starting program: /path/to/hello
...
Thread 2 hit Breakpoint 1, 0x0000000000001fd9 in main ()
(gdb) s
Single stepping until exit from function main,
which has no line number information.
Hello, world!
[Inferior 1 (process 7485) exited normally]
</code></pre></div></div>Sergey MelnychukOnce I was wondering, how different the ‘hello world’ in assembler on Linux would be from one on MacOS.Low-level TCP server in Rust with MIO2019-08-01T22:25:42+00:002019-08-01T22:25:42+00:00https://sergey-melnychuk.github.io/2019/08/01/rust-mio-tcp-server<p>It is time to get acquainted with <a href="https://github.com/tokio-rs/mio">Metal IO</a>, low-level cross-platform
abstraction over epoll/kqueue written in Rust.</p>
<p>TL;DR: <a href="https://github.com/sergey-melnychuk/mio-tcp-server">github</a>.</p>
<p>In this article I will show and explain how to write simple single-threaded
asynchronous TCP server, then teach it to mock HTTP protocol, and then
benchmark it with <code class="language-plaintext highlighter-rouge">ab</code>/<code class="language-plaintext highlighter-rouge">wrk</code>. The results are about to be impressive.</p>
<h3 id="getting-started">Getting started</h3>
<p>I’m using <code class="language-plaintext highlighter-rouge">mio = "0.6"</code>.</p>
<p>First, TCP listener is required:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">address</span> <span class="o">=</span> <span class="s">"0.0.0.0:8080"</span><span class="p">;</span>
<span class="k">let</span> <span class="n">listener</span> <span class="o">=</span> <span class="nn">TcpListener</span><span class="p">::</span><span class="nf">bind</span><span class="p">(</span><span class="o">&</span><span class="n">address</span><span class="nf">.parse</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">())</span><span class="nf">.unwrap</span><span class="p">();</span>
</code></pre></div></div>
<p>Then create <code class="language-plaintext highlighter-rouge">Poll</code> object and register listener at <code class="language-plaintext highlighter-rouge">Token(0)</code> for readable events,
activated by edge (not by level). More on <a href="https://en.wikipedia.org/wiki/Epoll#Triggering_modes">edge vs level</a>.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">poll</span> <span class="o">=</span> <span class="nn">Poll</span><span class="p">::</span><span class="nf">new</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="n">poll</span><span class="nf">.register</span><span class="p">(</span>
<span class="o">&</span><span class="n">listener</span><span class="p">,</span>
<span class="nf">Token</span><span class="p">(</span><span class="mi">0</span><span class="p">),</span>
<span class="nn">Ready</span><span class="p">::</span><span class="nf">readable</span><span class="p">(),</span>
<span class="nn">PollOpt</span><span class="p">::</span><span class="nf">edge</span><span class="p">())</span><span class="nf">.unwrap</span><span class="p">();</span>
</code></pre></div></div>
<p>The next essential part is to create <code class="language-plaintext highlighter-rouge">Events</code> object (of given capacity) and a
main loop (endless in this case). In the loop the events are polled and processed
one by one.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="k">mut</span> <span class="n">events</span> <span class="o">=</span> <span class="nn">Events</span><span class="p">::</span><span class="nf">with_capacity</span><span class="p">(</span><span class="mi">1024</span><span class="p">);</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="n">poll</span><span class="nf">.poll</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">events</span><span class="p">,</span> <span class="nb">None</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="k">for</span> <span class="n">event</span> <span class="n">in</span> <span class="o">&</span><span class="n">events</span> <span class="p">{</span>
<span class="c">// handle the event</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h3 id="accepting-connections-and-dropping-them">Accepting connections (and dropping them)</h3>
<p>The event can be one of the following:</p>
<ul>
<li>readable event on the listener means that there are incoming connections that
are ready to be accepted</li>
<li>event on the connected socket
<ul>
<li>readable - the socket has data available for reading</li>
<li>writable - the socket is ready for writing some data into it</li>
</ul>
</li>
</ul>
<p>The listener vs socket event can be distinguished by token, where for the listener
token is always zero, as it was registered in <code class="language-plaintext highlighter-rouge">Poll</code>.</p>
<p>Below is the simplest event handling approach, accept all incoming connections
in the loop, and for each connection - simply drop the socket. This will close
the connection. <a href="https://en.wikipedia.org/wiki/Discard_Protocol">Discard Protocol</a> at your service!</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// handle the event</span>
<span class="k">match</span> <span class="n">event</span><span class="nf">.token</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Token</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">match</span> <span class="n">listener</span><span class="nf">.accept</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Ok</span><span class="p">((</span><span class="n">socket</span><span class="p">,</span> <span class="n">address</span><span class="p">))</span> <span class="k">=></span> <span class="p">{</span>
<span class="c">// What to do with the connection?</span>
<span class="c">// One option is to simply drop it!</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Got connection from {}"</span><span class="p">,</span> <span class="n">address</span><span class="p">);</span>
<span class="p">},</span>
<span class="nf">Err</span><span class="p">(</span><span class="k">ref</span> <span class="n">e</span><span class="p">)</span> <span class="k">if</span> <span class="n">e</span><span class="nf">.kind</span><span class="p">()</span> <span class="o">==</span> <span class="nn">io</span><span class="p">::</span><span class="nn">ErrorKind</span><span class="p">::</span><span class="n">WouldBlock</span> <span class="k">=></span>
<span class="c">// No more connections ready to be accepted </span>
<span class="k">break</span><span class="p">,</span>
<span class="nf">Err</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="k">=></span>
<span class="nd">panic!</span><span class="p">(</span><span class="s">"Unexpected error: {}"</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="mi">_</span> <span class="k">=></span> <span class="p">()</span> <span class="c">// Ignore all other tokens</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Listener’s <code class="language-plaintext highlighter-rouge">.accept()</code> returns <code class="language-plaintext highlighter-rouge">std::io::Result<(TcpStream, SocketAddr)></code>
(as described <a href="https://docs.rs/mio/0.5.1/mio/tcp/struct.TcpListener.html#method.accept">here</a>), so I have to match and process successful
response or an error. But there is special kind of error
<code class="language-plaintext highlighter-rouge">io::ErrorKind::WouldBlock</code> (<a href="https://doc.rust-lang.org/nightly/std/io/enum.ErrorKind.html#variant.WouldBlock">doc</a>), that’s basically saying “I’m
about to wait (block) to make any progress”. This is the essential of
non-blocking behaviour - the point is just not to block (but return respective
error)! When encountered, such error simply means that there are no more incoming
connections waiting to be accepted at this point, so the loop is broken, and next
events get processed.</p>
<p>Now if I run the server and try connecting to it, I can see Discard Protocol in
action! Amazing right?!</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nc 127.0.0.1 8080
$
</code></pre></div></div>
<h3 id="registering-connections-for-events">Registering connections for events</h3>
<p>Speaking of next events. In order for next events to occur, the token-socket pair must be registered
with the <code class="language-plaintext highlighter-rouge">Poll</code> first. Under the hood, <code class="language-plaintext highlighter-rouge">Poll</code> keeps track which token corresponds to
which socket, but client code only gets access to token. This means if the server
intends to actually communicate with the client (and I’m pretty sure most servers
do), then token-socket pair must be stored somehow. In this example, I’m using
simple <code class="language-plaintext highlighter-rouge">HashMap<Token, TcpStream></code>, but <a href="https://docs.rs/slab/0.4.2/slab/">slab</a> might be more efficient
to use here.</p>
<p>Token is just a wrapper over <code class="language-plaintext highlighter-rouge">usize</code>, so simple counter is enough to provide
increasing sequence of tokens. Once socket is registered with respective token,
it is inserted into the HashMap.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="k">mut</span> <span class="n">counter</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">sockets</span><span class="p">:</span> <span class="n">HashMap</span><span class="o"><</span><span class="n">Token</span><span class="p">,</span> <span class="n">TcpStream</span><span class="o">></span> <span class="o">=</span> <span class="nn">HashMap</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
<span class="c">// handle the event</span>
<span class="k">match</span> <span class="n">event</span><span class="nf">.token</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Token</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">match</span> <span class="n">listener</span><span class="nf">.accept</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Ok</span><span class="p">((</span><span class="n">socket</span><span class="p">,</span> <span class="mi">_</span><span class="p">))</span> <span class="k">=></span> <span class="p">{</span>
<span class="n">counter</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">let</span> <span class="n">token</span> <span class="o">=</span> <span class="nf">Token</span><span class="p">(</span><span class="n">counter</span><span class="p">);</span>
<span class="c">// Register for readable events</span>
<span class="n">poll</span><span class="nf">.register</span><span class="p">(</span><span class="o">&</span><span class="n">socket</span><span class="p">,</span> <span class="n">token</span>
<span class="nn">Ready</span><span class="p">::</span><span class="nf">readable</span><span class="p">(),</span>
<span class="nn">PollOpt</span><span class="p">::</span><span class="nf">edge</span><span class="p">())</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="n">sockets</span><span class="nf">.insert</span><span class="p">(</span><span class="n">token</span><span class="p">,</span> <span class="n">socket</span><span class="p">);</span>
<span class="p">},</span>
<span class="nf">Err</span><span class="p">(</span><span class="k">ref</span> <span class="n">e</span><span class="p">)</span> <span class="k">if</span> <span class="n">e</span><span class="nf">.kind</span><span class="p">()</span> <span class="o">==</span> <span class="nn">io</span><span class="p">::</span><span class="nn">ErrorKind</span><span class="p">::</span><span class="n">WouldBlock</span> <span class="k">=></span>
<span class="c">// No more connections ready to be accepted </span>
<span class="k">break</span><span class="p">,</span>
<span class="nf">Err</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="k">=></span>
<span class="nd">panic!</span><span class="p">(</span><span class="s">"Unexpected error: {}"</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="n">token</span> <span class="k">if</span> <span class="n">event</span><span class="nf">.readiness</span><span class="p">()</span><span class="nf">.is_readable</span><span class="p">()</span> <span class="k">=></span> <span class="p">{</span>
<span class="c">// Socket associated with token is ready for reading data from it</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h3 id="reading-data-from-client">Reading data from client</h3>
<p>When readable event occurs for given token, this means data is ready to be read
from respective socket. I will use just array of bytes as a buffer for reading the data.</p>
<p>Read is performed in the loop until known <code class="language-plaintext highlighter-rouge">WouldBlock</code> error is returned. Each
call to read returns (if successful) actual number of bytes read, and when there
are zero bytes read - this <a href="https://doc.rust-lang.org/nightly/std/io/trait.Read.html#tymethod.read">means</a> client has disconnected already, and there is no
point if keeping the socket around (nor continuing the reading loop).</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Fixed size buffer for reading/writing to/from sockets</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">buffer</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span> <span class="k">as</span> <span class="nb">u8</span><span class="p">;</span> <span class="mi">1024</span><span class="p">];</span>
<span class="o">...</span>
<span class="n">token</span> <span class="k">if</span> <span class="n">event</span><span class="nf">.readiness</span><span class="p">()</span><span class="nf">.is_readable</span><span class="p">()</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">read</span> <span class="o">=</span> <span class="n">sockets</span><span class="nf">.get_mut</span><span class="p">(</span><span class="n">token</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">()</span><span class="nf">.read</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">buffer</span><span class="p">);</span>
<span class="k">match</span> <span class="n">read</span> <span class="p">{</span>
<span class="nf">Ok</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="c">// Successful read of zero bytes means connection is closed</span>
<span class="n">sockets</span><span class="nf">.remove</span><span class="p">(</span><span class="n">token</span><span class="p">);</span>
<span class="k">break</span><span class="p">;</span>
<span class="p">},</span>
<span class="nf">Ok</span><span class="p">(</span><span class="n">len</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="c">// Now do something with &buffer[0..len]</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Read {} bytes for token {}"</span><span class="p">,</span> <span class="n">len</span><span class="p">,</span> <span class="n">token</span><span class="na">.0</span><span class="p">);</span>
<span class="p">},</span>
<span class="nf">Err</span><span class="p">(</span><span class="k">ref</span> <span class="n">e</span><span class="p">)</span> <span class="k">if</span> <span class="n">e</span><span class="nf">.kind</span><span class="p">()</span> <span class="o">==</span> <span class="nn">io</span><span class="p">::</span><span class="nn">ErrorKind</span><span class="p">::</span><span class="n">WouldBlock</span> <span class="k">=></span> <span class="k">break</span><span class="p">,</span>
<span class="nf">Err</span><span class="p">(</span><span class="n">e</span><span class="p">)</span> <span class="k">=></span> <span class="nd">panic!</span><span class="p">(</span><span class="s">"Unexpected error: {}"</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="o">...</span>
</code></pre></div></div>
<h3 id="writing-data-to-the-client">Writing data to the client</h3>
<p>For a token to receive writable event, it must be registered in <code class="language-plaintext highlighter-rouge">Poll</code> first. The
<code class="language-plaintext highlighter-rouge">oneshot</code> option might be useful for scheduling writable events, this option makes
sure that event of interest is fired only once.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">poll</span><span class="nf">.register</span><span class="p">(</span><span class="o">&</span><span class="n">socket</span><span class="p">,</span> <span class="n">token</span>
<span class="nn">Ready</span><span class="p">::</span><span class="nf">writable</span><span class="p">(),</span>
<span class="nn">PollOpt</span><span class="p">::</span><span class="nf">edge</span><span class="p">()</span> <span class="p">|</span> <span class="nn">PollOpt</span><span class="p">::</span><span class="nf">oneshot</span><span class="p">())</span><span class="nf">.unwrap</span><span class="p">();</span>
</code></pre></div></div>
<p>Writing data to the client socket is similar, and is done via buffer as well, but
no explicit loop is required, as there is already a <a href="https://doc.rust-lang.org/std/io/trait.Write.html#method.write_all">method</a> that
is doing the loop: <code class="language-plaintext highlighter-rouge">write_all()</code>.</p>
<p>If I want all my protocol to do is to return how many bytes were received, I will
need an actual number of bytes written (<code class="language-plaintext highlighter-rouge">HashMap</code> will do), count bytes when
readable event occurs, then schedule a one-shot writable event, and when writable
event occurs - send the response and drop the connection.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="k">mut</span> <span class="n">response</span><span class="p">:</span> <span class="n">HashMap</span><span class="o"><</span><span class="n">Token</span><span class="p">,</span> <span class="nb">usize</span><span class="o">></span> <span class="o">=</span> <span class="nn">HashMap</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
<span class="o">...</span>
<span class="n">token</span> <span class="k">if</span> <span class="n">event</span><span class="nf">.readiness</span><span class="p">()</span><span class="nf">.is_readable</span><span class="p">()</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">bytes_read</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="o">...</span> <span class="c">// sum up number of bytes received</span>
<span class="p">}</span>
<span class="n">response</span><span class="nf">.insert</span><span class="p">(</span><span class="n">token</span><span class="p">,</span> <span class="n">bytes_read</span><span class="p">);</span>
<span class="c">// re-register for one-shot writable event</span>
<span class="p">}</span>
<span class="o">...</span>
<span class="n">token</span> <span class="k">if</span> <span class="n">event</span><span class="nf">.readiness</span><span class="p">()</span><span class="nf">.is_writable</span><span class="p">()</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">n_bytes</span> <span class="o">=</span> <span class="n">response</span><span class="p">[</span><span class="o">&</span><span class="n">token</span><span class="p">];</span>
<span class="k">let</span> <span class="n">message</span> <span class="o">=</span> <span class="nd">format!</span><span class="p">(</span><span class="s">"Received {} bytes</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">n_bytes</span><span class="p">);</span>
<span class="n">sockets</span><span class="nf">.get_mut</span><span class="p">(</span><span class="o">&</span><span class="n">token</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">()</span><span class="nf">.write_all</span><span class="p">(</span><span class="n">message</span><span class="nf">.as_bytes</span><span class="p">())</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="n">response</span><span class="nf">.remove</span><span class="p">(</span><span class="o">&</span><span class="n">token</span><span class="p">);</span>
<span class="n">sockets</span><span class="nf">.remove</span><span class="p">(</span><span class="o">&</span><span class="n">token</span><span class="p">);</span> <span class="c">// Drop the connection</span>
<span class="p">},</span>
</code></pre></div></div>
<h3 id="what-happens-between-reading-and-writing-data">What happens between reading and writing data?</h3>
<p>At this point I have reading the data from the socket, and (possible) writing
of the data to the socket. Yet writing never happens, as no token gets registered
for writable events!</p>
<p>When should I register a token for a writable event? Well, when it has something
to write! Sounds simple, isn’t it? In practice this means it’s time to actually
implement some <em>protocol</em>.</p>
<h3 id="how-do-i-implement-a-protocol">How do I implement a protocol?</h3>
<p>OK, I just want to send text (or JSON) back, and TCP is a <a href="https://en.wikipedia.org/wiki/Communication_protocol">protocol</a>,
why do I need more? Well, <a href="https://ru.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a> <em>is</em> a protocol, the Transmission Control
Protocol, the transport-level one. TCP cares about receiver to receive exact
amount of bytes in exact order that sender sent! So at the transport level, I have
to deal with two streams of bytes: one going from client to the server, and another
one going straight back.</p>
<p>What’s useful when dealing with servers, is an application level protocol (say, HTTP).
The application level protocol can define such entities as <code class="language-plaintext highlighter-rouge">request</code>, that server
receives from the client, and <code class="language-plaintext highlighter-rouge">response</code>, that client receives back from the server.</p>
<p>It is important to mention, that implementing HTTP <em>correctly</em> is not as easy as
it sounds, and even more, it is already done for you, e.g. in <a href="https://github.com/hyperium/hyper">hyper</a>.
Here I won’t bother with actually implementing HTTP, what I’m going to do instead
is to make my server behave as if it really understands GET requests, but will
always respond to such request with a response containing 6 bytes: <code class="language-plaintext highlighter-rouge">b"hello\n"</code>.</p>
<h3 id="mocking-http">Mocking HTTP</h3>
<p>For the sake of this article, mocking of HTTP is more than enough. I will use the
fact that HTTP request header is separated from request body (if any) with 4 bytes,
<code class="language-plaintext highlighter-rouge">b"\r\n\r\n"</code>. So if I keep track of what current client have sent, and if at any point
there are target 4 bytes in there, I can respond with pre-defined HTTP response:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HTTP/1.1 200 OK
Content-Type: text/html
Connection: keep-alive
Content-Length: 6
hello
</code></pre></div></div>
<p>Simple <code class="language-plaintext highlighter-rouge">HashMap</code> is enough to keep track of all the bytes received:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="k">mut</span> <span class="n">requests</span><span class="p">:</span> <span class="n">HashMap</span><span class="o"><</span><span class="n">Token</span><span class="p">,</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span> <span class="o">=</span> <span class="nn">HashMap</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
</code></pre></div></div>
<p>Once reading is done, it makes sense to check if request is ready:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">is_double_crnl</span><span class="p">(</span><span class="n">window</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">])</span> <span class="k">-></span> <span class="nb">bool</span> <span class="p">{</span> <span class="cm">/* trivial */</span> <span class="p">}</span>
<span class="k">let</span> <span class="n">ready</span> <span class="o">=</span> <span class="n">requests</span><span class="nf">.get</span><span class="p">(</span><span class="o">&</span><span class="n">token</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="nf">.windows</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="nf">.find</span><span class="p">(|</span><span class="n">window</span><span class="p">|</span> <span class="nf">is_double_crnl</span><span class="p">(</span><span class="o">*</span><span class="n">window</span><span class="p">))</span>
<span class="nf">.is_some</span><span class="p">();</span>
</code></pre></div></div>
<p>And if it is, it’s time to schedule some writing!</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="n">ready</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">socket</span> <span class="o">=</span> <span class="n">sockets</span><span class="nf">.get</span><span class="p">(</span><span class="o">&</span><span class="n">token</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="n">poll</span><span class="nf">.reregister</span><span class="p">(</span>
<span class="n">socket</span><span class="p">,</span>
<span class="n">token</span><span class="p">,</span>
<span class="nn">Ready</span><span class="p">::</span><span class="nf">writable</span><span class="p">(),</span>
<span class="nn">PollOpt</span><span class="p">::</span><span class="nf">edge</span><span class="p">()</span> <span class="p">|</span> <span class="nn">PollOpt</span><span class="p">::</span><span class="nf">oneshot</span><span class="p">())</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>After writing is completed, it is important to keep the connection open, and
<code class="language-plaintext highlighter-rouge">reregister</code> socket for reading again.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">poll</span><span class="nf">.reregister</span><span class="p">(</span>
<span class="n">sockets</span><span class="nf">.get</span><span class="p">(</span><span class="o">&</span><span class="n">token</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">(),</span>
<span class="n">token</span><span class="p">,</span>
<span class="nn">Ready</span><span class="p">::</span><span class="nf">readable</span><span class="p">(),</span>
<span class="nn">PollOpt</span><span class="p">::</span><span class="nf">edge</span><span class="p">())</span><span class="nf">.unwrap</span><span class="p">();</span>
</code></pre></div></div>
<p>Server is ready!</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl localhost:8080
hello
</code></pre></div></div>
<p>So the fun can start - let’s try an see how this <em>single-threaded</em>
server is performing. I will use common tools: <code class="language-plaintext highlighter-rouge">ab</code> and <code class="language-plaintext highlighter-rouge">wrk</code>. Note:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">ab</code> requires <code class="language-plaintext highlighter-rouge">-k</code> option to use <code class="language-plaintext highlighter-rouge">keep-alive</code> and reuse existing connections</li>
<li><code class="language-plaintext highlighter-rouge">wrk2</code> is actually used as <code class="language-plaintext highlighter-rouge">wrk</code>, thus <code class="language-plaintext highlighter-rouge">--rate</code> parameter is required</li>
<li><code class="language-plaintext highlighter-rouge">ab</code>/<code class="language-plaintext highlighter-rouge">wrk</code> is running on different VM than the server (but in the same region)</li>
</ul>
<p>Here are the numbers I got when trying benchmarking the server on the instance
<code class="language-plaintext highlighter-rouge">n1-standard-8 (8 vCPUs, 30 GB memory)</code> of some cloud provider
(that I’m not really allowed to mention here).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ab -n 1000000 -c 128 -k http://instance-1:8080/
<snip>
Requests per second: 105838.76 [#/sec] (mean)
Transfer rate: 9095.52 [Kbytes/sec] received
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ wrk -d 60s -t 8 -c 128 --rate 150k http://instance-1:8080/
<snip>
Requests/sec: 120596.75
Transfer/sec: 10.12MB
</code></pre></div></div>
<p>105k and 120k rps, not bad for a single thread.</p>
<p>Of course this can be qualified by cheating, but as long as real network is involved
(even inside the same zone), this is a real server under load, which can be
(more or less) meaningful bottom line for how fast networking can be done using single thread.</p>
<p>The full and runnable code is available on <a href="https://github.com/sergey-melnychuk/mio-tcp-server">github</a>, organized by one
logical chapter per pull-request:</p>
<ul>
<li>init project: <a href="https://github.com/sergey-melnychuk/mio-tcp-server/pull/1">PR#1</a></li>
<li>accept & discard: <a href="https://github.com/sergey-melnychuk/mio-tcp-server/pull/2">PR#2</a></li>
<li>read from socket: <a href="https://github.com/sergey-melnychuk/mio-tcp-server/pull/3">PR#3</a></li>
<li>writing to socket: <a href="https://github.com/sergey-melnychuk/mio-tcp-server/pull/4">PR#4</a></li>
<li>mocking HTTP: <a href="https://github.com/sergey-melnychuk/mio-tcp-server/pull/5">PR#5</a></li>
</ul>
<h3 id="where-to-go-from-here">Where to go from here</h3>
<p>Scaling to multiple threads: start <a href="https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/">here</a>.</p>Sergey MelnychukIt is time to get acquainted with Metal IO, low-level cross-platform abstraction over epoll/kqueue written in Rust.World Clock in Rust2019-07-24T20:24:42+00:002019-07-24T20:24:42+00:00https://sergey-melnychuk.github.io/2019/07/24/rust-world-clock<p>Just simple console app that asks for a time zone and prints current time in provided timezone (or in local if no time zone was provided).</p>
<p>It requires two dependencies in <code class="language-plaintext highlighter-rouge">Cargo.toml</code>: <a href="https://github.com/chronotope/chrono">chrono</a> and <a href="https://github.com/chronotope/chrono-tz">chrono-tz</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>chrono = "0.4"
chrono-tz = "0.5"
</code></pre></div></div>
<p>The app just runs in a loop (Ctrl-C to break), asks for a time zone, and prints time. That’s it!</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">extern</span> <span class="n">crate</span> <span class="n">chrono</span><span class="p">;</span>
<span class="k">extern</span> <span class="n">crate</span> <span class="n">chrono_tz</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="n">io</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">io</span><span class="p">::</span><span class="n">BufRead</span><span class="p">;</span>
<span class="k">use</span> <span class="nn">chrono</span><span class="p">::{</span><span class="n">Local</span><span class="p">,</span> <span class="n">Utc</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">chrono_tz</span><span class="p">::</span><span class="n">Tz</span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">stdin</span> <span class="o">=</span> <span class="nn">io</span><span class="p">::</span><span class="nf">stdin</span><span class="p">();</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">line</span> <span class="o">=</span> <span class="n">stdin</span><span class="nf">.lock</span><span class="p">()</span><span class="nf">.lines</span><span class="p">()</span><span class="nf">.next</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="k">if</span> <span class="n">line</span><span class="nf">.is_empty</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">now</span> <span class="o">=</span> <span class="nn">Local</span><span class="p">::</span><span class="nf">now</span><span class="p">();</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span> <span class="n">now</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">match</span> <span class="n">line</span><span class="py">.parse</span><span class="p">::</span><span class="o"><</span><span class="n">Tz</span><span class="o">></span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Ok</span><span class="p">(</span><span class="n">tz</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">now</span> <span class="o">=</span> <span class="nn">Utc</span><span class="p">::</span><span class="nf">now</span><span class="p">();</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span> <span class="n">now</span><span class="nf">.with_timezone</span><span class="p">(</span><span class="o">&</span><span class="n">tz</span><span class="p">));</span>
<span class="p">},</span>
<span class="nf">Err</span><span class="p">(</span><span class="mi">_</span><span class="p">)</span> <span class="k">=></span> <span class="nd">println!</span><span class="p">(</span><span class="s">"(no such timezone)"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>For more details on date & time: <a href="https://rust-lang-nursery.github.io/rust-cookbook/datetime/parse.html">cookbook</a>.</p>Sergey MelnychukJust simple console app that asks for a time zone and prints current time in provided timezone (or in local if no time zone was provided).Setting up SSH reverse proxy2019-07-09T21:30:42+00:002019-07-09T21:30:42+00:00https://sergey-melnychuk.github.io/2019/07/09/ssh-proxy<p>Providing an SSH access to a home server (usually hidden behind NAT) from outside world (Internet) sounds like a tricky task to configure. But it is not, literally 2 commands and a VPS witn public IP is enough to do that.</p>
<p>I will need following:</p>
<ul>
<li>Instances
<ul>
<li>VPS with public <code class="language-plaintext highlighter-rouge">$IP</code></li>
<li><code class="language-plaintext highlighter-rouge">home</code> server</li>
<li><code class="language-plaintext highlighter-rouge">away</code> client</li>
</ul>
</li>
<li>SSH keys
<ul>
<li><code class="language-plaintext highlighter-rouge">tunnel_rsa</code> to access VPS from <code class="language-plaintext highlighter-rouge">home</code></li>
<li><code class="language-plaintext highlighter-rouge">key_rsa</code> to access <code class="language-plaintext highlighter-rouge">home</code> from <code class="language-plaintext highlighter-rouge">away</code></li>
</ul>
</li>
</ul>
<p>On <code class="language-plaintext highlighter-rouge">home</code> instance, this command connects to <code class="language-plaintext highlighter-rouge">$IP</code> (VPS) and spawns there a listening at port 2222 reverse proxy connection back to <code class="language-plaintext highlighter-rouge">home</code>. This can run in background.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># home</span>
<span class="nv">$ </span>ssh <span class="nt">-i</span> ~/.ssh/tunnel_rsa <span class="nt">-R</span> 0.0.0.0:2222:localhost:22 <span class="nv">$IP</span> <span class="nt">-N</span>
</code></pre></div></div>
<p>Now from <code class="language-plaintext highlighter-rouge">away</code> instance, SSH-ing to reverse proxy connection will let us to connect to <code class="language-plaintext highlighter-rouge">home</code> instance. Easy and securely.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># away</span>
<span class="nv">$ </span>ssh <span class="nt">-i</span> ~/.ssh/key_rsa key@<span class="nv">$IP</span> <span class="nt">-p</span> 2222
</code></pre></div></div>
<p>Nice!</p>
<p>UPDATE:</p>
<p>Some useful <a href="https://unix.stackexchange.com/a/17836">tip</a> on configuration <code class="language-plaintext highlighter-rouge">/etc/ssh/sshd</code> on VPC instance:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>AllowTcpForwarding yes
TCPKeepAlive yes
</code></pre></div></div>
<p>Explanation of sshd <code class="language-plaintext highlighter-rouge">-R</code> <a href="https://superuser.com/a/591963">flag</a>.</p>Sergey MelnychukProviding an SSH access to a home server (usually hidden behind NAT) from outside world (Internet) sounds like a tricky task to configure. But it is not, literally 2 commands and a VPS witn public IP is enough to do that.Distributed Membership Protocol2019-05-12T17:30:42+00:002019-05-12T17:30:42+00:00https://sergey-melnychuk.github.io/2019/05/12/distributed-membership-protocol<p>Once I had a long winter evening to spare, and I was thinking, why don’t I implement a distributed membership and failure-detection protocol in Rust?</p>
<p>TL;DR: <a href="https://github.com/sergey-melnychuk/gossip-peer">github</a></p>
<p>At the moment, it is rather example than usable library, but I have great plans for making it into efficient and usable tool for decentralized coordination.</p>
<p>Being already pretty <a href="https://github.com/sergey-melnychuk/distributed-algorithms">familiar</a> with distributed membership and failure detection protocols, goal was mostly to write some Rust code and to get familiar with <code class="language-plaintext highlighter-rouge">std::net</code> package. The UDP protocol support is rather straightforward:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">socket</span> <span class="o">=</span> <span class="nn">UdpSocket</span><span class="p">::</span><span class="nf">bind</span><span class="p">(</span><span class="nn">SocketAddr</span><span class="p">::</span><span class="nf">from</span><span class="p">(([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span> <span class="n">port</span><span class="p">)))</span>
<span class="nf">.expect</span><span class="p">(</span><span class="s">"failed to bind"</span><span class="p">);</span>
<span class="n">socket</span><span class="nf">.set_read_timeout</span><span class="p">(</span><span class="nf">Some</span><span class="p">(</span><span class="nn">Duration</span><span class="p">::</span><span class="nf">from_millis</span><span class="p">(</span><span class="n">read_timeout_millis</span><span class="p">)))</span>
<span class="nf">.expect</span><span class="p">(</span><span class="s">"failed to set read timeout"</span><span class="p">);</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">buf</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0u8</span><span class="p">;</span> <span class="mi">1024</span><span class="p">];</span>
<span class="k">loop</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">res</span> <span class="o">=</span> <span class="n">socket</span><span class="nf">.recv_from</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">buf</span><span class="p">);</span>
<span class="k">if</span> <span class="n">res</span><span class="nf">.is_ok</span><span class="p">()</span> <span class="p">{</span> <span class="cm">/* do something useful */</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The idea of distributed membership and failure detection is simple - if a node doesn’t hear from peer for some time, such peer can be considered as failed. Thus for a peer to not be considered as failed, it must spread it’s good and healthy status to other peers. I decided to start with all-to-all heartbeat propagation and with constant timeouts for marking peers as unreachable or failed. <a href="https://www.consul.io/docs/internals/gossip.html">Gossip protocol</a> is a well defined concept after all.</p>
<p>The structure of a peer is very simple, and keeps address, most recent received heartbeat and timestamp when this most recent heartbeat was received. This timestamp is used later to reason about peer state: if’s too old, then reasonable assumption is made that such peer has failed.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[derive(Debug,</span> <span class="nd">Copy,</span> <span class="nd">Clone)]</span>
<span class="k">pub</span> <span class="k">struct</span> <span class="n">Record</span> <span class="p">{</span>
<span class="k">pub</span> <span class="n">addr</span><span class="p">:</span> <span class="n">Addr</span><span class="p">,</span>
<span class="k">pub</span> <span class="n">beat</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span>
<span class="k">pub</span> <span class="n">time</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The protocol itself is very simple as well, and consist of only two messages:</p>
<ul>
<li><em>Join</em> is sent to one of peers when newly started node want’s to join existing cluster</li>
<li><em>List</em> is sent to share state of node’s peers with those peers</li>
</ul>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[derive(Debug)]</span>
<span class="k">pub</span> <span class="k">enum</span> <span class="n">Message</span> <span class="p">{</span>
<span class="nf">Join</span><span class="p">(</span><span class="n">Record</span><span class="p">),</span>
<span class="nf">List</span><span class="p">(</span><span class="nb">Vec</span><span class="o"><</span><span class="n">Record</span><span class="o">></span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Sending list of all peers to all peers does sound like an overhead, and it is. So for large deployments it makes sense to actually apply ‘gossip’ approach for sharing state, sending only fraction of peers’ states (say, 3) to other peers. Such infectious gossip will converge in linear time (can’t remember the proof link, just use DuckDuckGo), keeping traffic linear as well (each peer sends constant-size messages). This will look much better, and will reduce quadratic (each peer sends to each) throughput and qubic (!) bandwith (each message contains all peers).</p>
<p>Such example of distributed membership can be run locally, after building with <code class="language-plaintext highlighter-rouge">caro build</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Terminal 1</span>
<span class="nv">$ </span>./gossip-peer 12000
listening at: 12000
<span class="c"># Now start another node in Terminal 2</span>
append: Record <span class="o">{</span> addr: 127.0.0.1:12000, beat: 0, <span class="nb">time</span>: <timestamp> <span class="o">}</span>
<span class="c"># Now stop this node by Ctrl-C</span>
^C
<span class="nv">$ </span><span class="c"># Now check Terminal 2 if it detected stopped node</span>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Terminal 2</span>
<span class="nv">$ </span>./gossip-peer 12001 127.0.0.1:12000
listening at: 12001
append: Record <span class="o">{</span> addr: 127.0.0.1:12001, beat: 1, <span class="nb">time</span>: <timestamp> <span class="o">}</span>
<span class="c"># Now check Terminal 1 - it must detect new node</span>
remove: Record <span class="o">{</span> addr: 127.0.0.1:12000, beat: 42, <span class="nb">time</span>: <timestamp> <span class="o">}</span>
<span class="c"># Unresponsive node detected and removed from list</span>
^C
<span class="nv">$ </span>
</code></pre></div></div>
<p>As for further plans of <a href="https://github.com/sergey-melnychuk/gossip-peer">gossip-peer</a> development, now I can imagine doing such (probably) useful things:</p>
<ul>
<li>Extensive unit- and system-test coverage :)</li>
<li>Switch from <code class="language-plaintext highlighter-rouge">std::net</code> to <a href="https://tokio.rs">tokio</a> or <a href="https://actix.rs">actix</a></li>
<li>Automatic switch from all-to-all to gossip when cluster reach certain size</li>
<li>Support adaptive timeout for failure detection to address network congestion condition</li>
<li>Support decentralized service discovery by adding <code class="language-plaintext highlighter-rouge">tag</code> property for each peer</li>
<li>Support decentralized load balancing among peers with the same <code class="language-plaintext highlighter-rouge">tag</code></li>
<li>Support multicast-based peers lookup (won’t work in the cloud though)</li>
<li>Implement application-level broadcast/multicast (can be used for cloud deployments)</li>
<li>Support simple distributed replicated in-memory key-value database (like <a href="https://www.consul.io">consul</a>)</li>
<li>Provide low-footprint library to include peer-awareness in existing network applications</li>
<li>Some other interesting distributed wheel to re-invent!</li>
</ul>Sergey MelnychukOnce I had a long winter evening to spare, and I was thinking, why don’t I implement a distributed membership and failure-detection protocol in Rust?Reactive Game of Life with Akka2019-04-19T16:56:42+00:002019-04-19T16:56:42+00:00https://sergey-melnychuk.github.io/2019/04/19/akka-reactive-game-of-life<p>Quite for some time I was thinking about refreshing Akka skills and providing some simple demo of what one can do with Akka and <a href="https://en.wikipedia.org/wiki/Actor_model">Actor Model</a>. Finally, the <del>winter</del> time has come, I will show the <a href="https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life">Conway’s Game of Life</a> reactive implementation with <a href="https://doc.akka.io/docs/akka/current/actors.html">Akka Actors</a>.</p>
<p>TL;DR: <a href="https://github.com/sergey-melnychuk/akka-reactive-game-of-life">code</a></p>
<p>The rules are very simple, the current state of a cell on a two-dimentional grid depends only on previous state of the cell and a number of “alive” cells among surrounding 8 neighbors cells (immediate vertical, horizontal and diagonal adjacent cells). The implementation in Scala looks like this:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">object</span> <span class="nc">Cell</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">apply</span><span class="o">(</span><span class="n">alive</span><span class="k">:</span> <span class="kt">Boolean</span><span class="o">,</span> <span class="n">around</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span><span class="k">:</span> <span class="kt">Boolean</span> <span class="o">=</span> <span class="o">{</span>
<span class="nf">assert</span><span class="o">(</span><span class="n">around</span> <span class="o">>=</span> <span class="mi">0</span><span class="o">)</span>
<span class="o">(</span><span class="n">alive</span><span class="o">,</span> <span class="n">around</span><span class="o">)</span> <span class="k">match</span> <span class="o">{</span>
<span class="nf">case</span> <span class="o">(</span><span class="kc">true</span><span class="o">,</span> <span class="n">n</span><span class="o">)</span> <span class="k">if</span> <span class="n">n</span> <span class="o"><</span> <span class="mi">2</span> <span class="k">=></span> <span class="kc">false</span>
<span class="nf">case</span> <span class="o">(</span><span class="kc">true</span><span class="o">,</span> <span class="n">n</span><span class="o">)</span> <span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">2</span> <span class="o">||</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">3</span> <span class="k">=></span> <span class="kc">true</span>
<span class="nf">case</span> <span class="o">(</span><span class="kc">true</span><span class="o">,</span> <span class="n">n</span><span class="o">)</span> <span class="k">if</span> <span class="n">n</span> <span class="o">></span> <span class="mi">3</span> <span class="k">=></span> <span class="kc">false</span>
<span class="nf">case</span> <span class="o">(</span><span class="kc">false</span><span class="o">,</span> <span class="n">n</span><span class="o">)</span> <span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">3</span> <span class="k">=></span> <span class="kc">true</span>
<span class="k">case</span> <span class="k">_</span> <span class="k">=></span> <span class="kc">false</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The initial idea I had was to keep each cell’s state in a dedicated actor, thus allowing “reactive” and consistent transitions to the next state. The only events happening are cells becoming “alive” or “empty”, thus once cell becomes “alive”, the surrounding 8 cells receive a message that number of neighbors increased, and respectfully, when cell becomes “empty” - decreased. Scaling this out (to suppot grids that don’t fit single instance) might be tricky, as messages between adjacent cells must still be able to find their way to addressed cell. <a href="https://doc.akka.io/docs/akka/current/general/remoting.html">Location transparency</a> FTW.</p>
<p>For this post, I will limit the scope to small grids, but the code doesn’t need to be changed much for a distributed case, <em>just</em> using <a href="https://doc.akka.io/docs/akka/current/remoting.html">Akka Remoting</a> looks enough. The distributed option will remain on my to-do list, as it might be interesting to run. Tricky to visualize/observe though - how to observe million by million grid? It’s 1E12 cells, but it might be interesting to play with some statistics about it.</p>
<p>Back to small grids, but keep scaling out in mind. I need to split a grid in chunks, perfect use case for a <a href="https://en.wikipedia.org/wiki/Quadtree">Quad Tree</a>. The quad tree nodes are <em>forks</em> and <em>leaves</em>, with <em>forks</em> keeping list of quads that belongs a level down, and <em>leaves</em> keeping the sub-grid of cells and mapping from cell’s position to a <code class="language-plaintext highlighter-rouge">CellActor</code>. So for sub-quad border-line cells, some adjacent cells lay in different quads, and it makes sense to forward updates to such “remote” cells via parent quad tree node. Then quad tree node can decide if an update should go to respective quad or go to a higher level of a quad tree. Eventually such event will land respective cell, and the longet such path (for central 4 cells) will go up to the quad tree root and then down to respective cell through quad tree nodes. It scales though, and each actor keeps doing their thing: <code class="language-plaintext highlighter-rouge">CellActor</code> stores state and handles updates from neighbors, <code class="language-plaintext highlighter-rouge">GridActor</code> stores either split quads or references to actual cells and handles routing of updates for “remote” cells.</p>
<p>This is all very nice, but how do I actually <em>see</em> what is really happening on the grid? I isolated 2 abstractions: <code class="language-plaintext highlighter-rouge">Contol</code> to let user provide some input (click on specific cell) and <code class="language-plaintext highlighter-rouge">Display</code>, that is responsible for rendering specific cell in a given state. I believe the signatures below are self-explanatory enough:</p>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// The position of the cell on the grid</span>
<span class="k">case</span> <span class="k">class</span> <span class="nc">Pos</span><span class="o">(</span><span class="n">row</span><span class="k">:</span> <span class="kt">Int</span><span class="o">,</span> <span class="n">col</span><span class="k">:</span> <span class="kt">Int</span><span class="o">)</span>
<span class="k">trait</span> <span class="nc">Control</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">click</span><span class="o">(</span><span class="n">pos</span><span class="k">:</span> <span class="kt">Pos</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span>
<span class="k">def</span> <span class="nf">terminate</span><span class="o">()</span><span class="k">:</span> <span class="kt">Unit</span>
<span class="o">}</span>
<span class="k">trait</span> <span class="nc">Display</span> <span class="o">{</span>
<span class="k">def</span> <span class="nf">fill</span><span class="o">(</span><span class="n">pos</span><span class="k">:</span> <span class="kt">Pos</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span>
<span class="k">def</span> <span class="nf">clear</span><span class="o">(</span><span class="n">pos</span><span class="k">:</span> <span class="kt">Pos</span><span class="o">)</span><span class="k">:</span> <span class="kt">Unit</span>
<span class="o">}</span>
</code></pre></div></div>
<p>These two abstractions are more than enough to keep the show going! The actuall execution of input and state updates is performed in scope of <code class="language-plaintext highlighter-rouge">DisplayActor</code>, as this is essentially all what can happen: user can click or a cell gets rendered on a screen.</p>
<p>The actual rendering is implemented via <a href="https://gitlab.com/h2b/SimGraf">SimGraf</a>, nice and small UI library (built on top of Akka as well). The result looks like this:</p>
<p><img src="https://sergey-melnychuk.github.io/assets/2019-04-19-akka-reactive-game-of-life/grid.png" alt="Conway's Game of Life" /></p>
<p>The unit-tests of the <a href="https://github.com/sergey-melnychuk/akka-reactive-game-of-life">code</a> are left as an excercise for a reader.</p>Sergey MelnychukQuite for some time I was thinking about refreshing Akka skills and providing some simple demo of what one can do with Akka and Actor Model. Finally, the winter time has come, I will show the Conway’s Game of Life reactive implementation with Akka Actors.Borrowed Byte Buffer in Rust2019-01-17T19:13:00+00:002019-01-17T19:13:00+00:00https://sergey-melnychuk.github.io/2019/01/17/borrowed-byte-buffer<p>It started with re-implementation of gossip-based distributed membership protocol, this time in Rust. I was looking for a fast and simple way to put data into array of bytes, and to get data from an array of bytes. Something like Java’s <a href="https://docs.oracle.com/javase/8/docs/api/java/nio/ByteBuffer.html">ByteBuffer</a>.</p>
<p>Playing with memory allocation is fun, but it was not what I was looking for. That’s why obvious decision was to read/write over an existing buffer, owned elsewhere. Apparently, this made excellent example of using Rust’s lifetime checks, as that exsisting buffer isn’t really owned by byte-buffer object but only referenced.</p>
<p>The full code for mutable (writable) and immutable (readable) borrowed byte buffer with unit-tests is under 300 LoC, awailable at <a href="https://github.com/sergey-melnychuk/borrowed-byte-buffer">github</a> and published at <a href="https://crates.io/crates/borrowed-byte-buffer">crates</a>. It’s clearly not for production use, hence the version <code class="language-plaintext highlighter-rouge">0.0.1</code>. Code is not even close to elegant nor idiomatic, but it gets the job done.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">struct</span> <span class="n">ByteBuf</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="n">pos</span><span class="p">:</span> <span class="nb">usize</span><span class="p">,</span> <span class="c">// keep track of the current position in the array</span>
<span class="n">buf</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="p">[</span><span class="nb">u8</span><span class="p">],</span> <span class="c">// reference to the array with lifetime parameter 'a</span>
<span class="p">}</span>
<span class="k">pub</span> <span class="k">struct</span> <span class="n">ByteBufMut</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="n">pos</span><span class="p">:</span> <span class="nb">usize</span><span class="p">,</span> <span class="c">// same</span>
<span class="n">buf</span><span class="p">:</span> <span class="o">&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="p">[</span><span class="nb">u8</span><span class="p">],</span> <span class="c">// mutable reference with lifetime 'a</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">ByteBuf</code> allows reading u8, u16, u32 and u64, as well as <code class="language-plaintext highlighter-rouge">ByteBufMut</code> allows writing respectively 1, 2, 4 and 8 bytes at once. Getting words to bytes is pretty straightforward application of bit arithmetic.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">impl</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="n">ByteBuf</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">get_u8</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="c">// return one byte from the buffer, if it is awailable</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">impl</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="n">ByteBufMut</span><span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="p">{</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">put_u8</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="k">self</span><span class="p">,</span> <span class="n">x</span><span class="p">:</span> <span class="nb">u8</span><span class="p">)</span> <span class="k">-></span> <span class="nb">usize</span> <span class="p">{</span>
<span class="c">// put one byte in the buffer and return number of bytes written</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>As long as there is no control/ownership over underlying buffer, the <code class="language-plaintext highlighter-rouge">get_*</code> methods return Option (reading after reaching end of buffer will return None) and <code class="language-plaintext highlighter-rouge">put_*</code> methods return actual number of bytes written to the buffer. It’s not too convenient for sure, but it is the only way to go without managing and owning the underlynig buffer.</p>
<p>As a direction for improvement, the generic struct (de)serialization to/from binary woud be nice. Though reinventing the wheel doesn’t make much sense when there are already awailable things like <a href="https://github.com/real-logic/simple-binary-encoding">simple binary encoding</a> and <a href="https://docs.serde.rs/serde/trait.Serializer.html">serde serialization</a></p>Sergey MelnychukIt started with re-implementation of gossip-based distributed membership protocol, this time in Rust. I was looking for a fast and simple way to put data into array of bytes, and to get data from an array of bytes. Something like Java’s ByteBuffer.Command prompt setup2018-06-03T15:56:00+00:002018-06-03T15:56:00+00:00https://sergey-melnychuk.github.io/2018/06/03/command-prompt-setup<p>Quite for some time I was taking command prompt for granted, thinking that it is just as it is. But today is the day, I’m setting up the command prompt that I like!</p>
<p>First thing to do was actually to decide what exact command prompt I like. After searching the Internet for details about setting up prompt, I have picked two links to put in this post: <a href="https://www.cyberciti.biz/tips/howto-linux-unix-bash-shell-setup-prompt.html">first</a> about general details for PS1 environment variable, and <a href="https://gist.github.com/githubteacher/e75edf29d76571f8cc6c">second</a> about putting current git branch to command prompt. For me it appears that ideal prompt is as easy as <code class="language-plaintext highlighter-rouge"><username>@[<directory> (:<git branch, if any>)] $ </code>. My final setup is published at github as a <a href="https://gist.github.com/sergey-melnychuk/89e6e78ac212dd4b55cc819356108459">gist</a>.</p>
<p>In short, this is how to detect current git branch (here I put prefix ` :` before branch name in order to completely remove branch section of command prompt if there aren’t any git branch - this happens, sometimes I don’t go to folders between git repositories, that have no git branch).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## Get output of 'git branch' and leave only row that contains '*'</span>
git branch 2> /dev/null | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'/^[^*]/d'</span> <span class="nt">-e</span> <span class="s1">'s/* \(.*\)/ :\1/'</span>
</code></pre></div></div>
<p>From <a href="https://www.cyberciti.biz/tips/howto-linux-unix-bash-shell-setup-prompt.html">article</a> I can find needed characters for username and current directory.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## \u - the username of the current user</span>
<span class="c">## \w - the current working directory, with $HOME abbreviated with a tilde</span>
</code></pre></div></div>
<p>Now it’s time to put everything together and get final result. Put this into <code class="language-plaintext highlighter-rouge">~/.bashrc</code> on Linux and/or <code class="language-plaintext highlighter-rouge">~/.bash_profile</code> on MacOS.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>get_git_branch<span class="o">()</span> <span class="o">{</span>
git branch 2> /dev/null | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'/^[^*]/d'</span> <span class="nt">-e</span> <span class="s1">'s/* \(.*\)/ :\1/'</span>
<span class="o">}</span>
<span class="nb">export </span><span class="nv">PS1</span><span class="o">=</span><span class="s1">'\u@[\w$(get_git_branch)] $ '</span>
</code></pre></div></div>
<p>I think I’ve never been typing words ‘command prompt’ so much before writing this post.</p>Sergey MelnychukQuite for some time I was taking command prompt for granted, thinking that it is just as it is. But today is the day, I’m setting up the command prompt that I like!Testing RabbitMQ Client in Java2018-05-27T13:35:00+00:002018-05-27T13:35:00+00:00https://sergey-melnychuk.github.io/2018/05/27/testing-rabbitmq-client-java<p>Writing RabbitMQ client in Java is very well described in the <a href="https://www.rabbitmq.com/tutorials/tutorial-one-java.html">tutorial</a> and client <a href="https://www.rabbitmq.com/java-client.html">docs</a>, but writing test-coverage for such client can be tricky.</p>
<p>It makes sense to test message queue client against real message queue, and it makes sense to pick the message queue broker that can be embedded into tests code. The answer I found is Apache <a href="https://qpid.apache.org/">QPID</a>.</p>
<p>Next in this post I will show how to set up embedded QPID broker and run send/receive test workload against it. Here I will provide the full solution without much comments (as it is extremely straightforward), but for more details like broker configuration etc the QPID <a href="https://qpid.apache.org/releases/qpid-java-6.0.7/java-broker/book/Java-Broker-Introduction.html">docs</a> is a good starting point.</p>
<p>This is how to add required Java dependencies using Gradle.</p>
<div class="language-groovy highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">compile</span><span class="o">(</span><span class="s1">'com.rabbitmq:amqp-client:5.0.0'</span><span class="o">)</span>
<span class="n">testCompile</span><span class="o">(</span><span class="s1">'org.apache.qpid:qpid-broker:6.1.4'</span><span class="o">)</span>
</code></pre></div></div>
<p>This is QPID broker configuration file that will be used for this example.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"broker"</span><span class="p">,</span><span class="w">
</span><span class="nl">"modelVersion"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"6.1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"accesscontrolproviders"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AllowAll"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AllowAll"</span><span class="p">,</span><span class="w">
</span><span class="nl">"priority"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">9999</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nl">"authenticationproviders"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"plain"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Plain"</span><span class="p">,</span><span class="w">
</span><span class="nl">"users"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"id"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"5c009963-eb49-4249-a150-833db543fe9e"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"managed"</span><span class="p">,</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"guest"</span><span class="p">,</span><span class="w">
</span><span class="nl">"password"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"guest"</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nl">"keystores"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"default"</span><span class="p">,</span><span class="w">
</span><span class="nl">"password"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"broker"</span><span class="p">,</span><span class="w">
</span><span class="nl">"storeUrl"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"qpid/brokerkeystore"</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nl">"ports"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AMQP"</span><span class="p">,</span><span class="w">
</span><span class="nl">"port"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"${qpid.amqp_port}"</span><span class="p">,</span><span class="w">
</span><span class="nl">"authenticationProvider"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"plain"</span><span class="p">,</span><span class="w">
</span><span class="nl">"keyStore"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"default"</span><span class="p">,</span><span class="w">
</span><span class="nl">"transports"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"SSL"</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nl">"virtualhostaliases"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"defaultAlias"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"defaultAlias"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"hostnameAlias"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"hostnameAlias"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"nameAlias"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"nameAlias"</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nl">"virtualhostnodes"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"default"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"JSON"</span><span class="p">,</span><span class="w">
</span><span class="nl">"defaultVirtualHostNode"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="p">,</span><span class="w">
</span><span class="nl">"virtualHostInitialConfiguration"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"${qpid.initial_config_virtualhost_config}"</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>For the sake of SSL support, configuration requires keystore. Useful links on keystores from <a href="https://www.digitalocean.com/community/tutorials/java-keytool-essentials-working-with-java-keystores">digitalocean</a> and <a href="https://docs.oracle.com/cd/E19509-01/820-3503/ggfen/index.html">oracle</a>. The keystore described in this post looks like this:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>keytool <span class="nt">-list</span> <span class="nt">-keystore</span> brokerkeystore
Enter keystore password:
Keystore <span class="nb">type</span>: JKS
Keystore provider: SUN
Your keystore contains 1 entry
broker, May 27, 2017, PrivateKeyEntry,
Certificate fingerprint <span class="o">(</span>SHA1<span class="o">)</span>: <removed>
</code></pre></div></div>
<p>Now the RabbitMQ client code:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">RabbitClient</span> <span class="kd">implements</span> <span class="nc">Supplier</span><span class="o"><</span><span class="nc">String</span><span class="o">>,</span> <span class="nc">Consumer</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Logger</span> <span class="n">logger</span> <span class="o">=</span> <span class="nc">LoggerFactory</span><span class="o">.</span><span class="na">getLogger</span><span class="o">(</span><span class="nc">RabbitClient</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">String</span> <span class="no">EXCHANGE_TYPE</span> <span class="o">=</span> <span class="s">"direct"</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">boolean</span> <span class="no">QUEUE_DURABLE</span> <span class="o">=</span> <span class="kc">true</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">boolean</span> <span class="no">QUEUE_EXCLUSIVE</span> <span class="o">=</span> <span class="kc">false</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="kt">boolean</span> <span class="no">QUEUE_AUTODELETE</span> <span class="o">=</span> <span class="kc">false</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">Config</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">String</span> <span class="n">host</span><span class="o">;</span>
<span class="kd">final</span> <span class="kt">int</span> <span class="n">port</span><span class="o">;</span>
<span class="kd">final</span> <span class="nc">String</span> <span class="n">virtualHost</span><span class="o">;</span>
<span class="kd">final</span> <span class="nc">String</span> <span class="n">username</span><span class="o">;</span>
<span class="kd">final</span> <span class="nc">String</span> <span class="n">password</span><span class="o">;</span>
<span class="kd">final</span> <span class="nc">String</span> <span class="n">queueName</span><span class="o">;</span>
<span class="kd">final</span> <span class="nc">String</span> <span class="n">exchangeName</span><span class="o">;</span>
<span class="kd">final</span> <span class="nc">String</span> <span class="n">routingKey</span><span class="o">;</span>
<span class="kd">final</span> <span class="kt">boolean</span> <span class="n">useSsl</span><span class="o">;</span>
<span class="kd">public</span> <span class="nf">Config</span><span class="o">(</span>
<span class="nc">String</span> <span class="n">host</span><span class="o">,</span>
<span class="kt">int</span> <span class="n">port</span><span class="o">,</span>
<span class="nc">String</span> <span class="n">virtualHost</span><span class="o">,</span>
<span class="nc">String</span> <span class="n">username</span><span class="o">,</span>
<span class="nc">String</span> <span class="n">password</span><span class="o">,</span>
<span class="nc">String</span> <span class="n">queueName</span><span class="o">,</span>
<span class="nc">String</span> <span class="n">exchangeName</span><span class="o">,</span>
<span class="nc">String</span> <span class="n">routingKey</span><span class="o">,</span>
<span class="kt">boolean</span> <span class="n">useSsl</span>
<span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">host</span> <span class="o">=</span> <span class="n">host</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">port</span> <span class="o">=</span> <span class="n">port</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">virtualHost</span> <span class="o">=</span> <span class="n">virtualHost</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">username</span> <span class="o">=</span> <span class="n">username</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">password</span> <span class="o">=</span> <span class="n">password</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">queueName</span> <span class="o">=</span> <span class="n">queueName</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">exchangeName</span> <span class="o">=</span> <span class="n">exchangeName</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">routingKey</span> <span class="o">=</span> <span class="n">routingKey</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">useSsl</span> <span class="o">=</span> <span class="n">useSsl</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">RabbitClient</span><span class="o">.</span><span class="na">Config</span> <span class="n">config</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="nc">ConnectionFactory</span> <span class="n">connectionFactory</span><span class="o">;</span>
<span class="kd">public</span> <span class="nf">RabbitClient</span><span class="o">(</span><span class="kd">final</span> <span class="nc">RabbitClient</span><span class="o">.</span><span class="na">Config</span> <span class="n">config</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">config</span> <span class="o">=</span> <span class="n">config</span><span class="o">;</span>
<span class="k">this</span><span class="o">.</span><span class="na">connectionFactory</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ConnectionFactory</span><span class="o">();</span>
<span class="k">this</span><span class="o">.</span><span class="na">connectionFactory</span><span class="o">.</span><span class="na">setUsername</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">username</span><span class="o">);</span>
<span class="k">this</span><span class="o">.</span><span class="na">connectionFactory</span><span class="o">.</span><span class="na">setPassword</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">password</span><span class="o">);</span>
<span class="k">this</span><span class="o">.</span><span class="na">connectionFactory</span><span class="o">.</span><span class="na">setVirtualHost</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">virtualHost</span><span class="o">);</span>
<span class="k">this</span><span class="o">.</span><span class="na">connectionFactory</span><span class="o">.</span><span class="na">setHost</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">host</span><span class="o">);</span>
<span class="k">this</span><span class="o">.</span><span class="na">connectionFactory</span><span class="o">.</span><span class="na">setPort</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">port</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">useSsl</span><span class="o">)</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">connectionFactory</span><span class="o">.</span><span class="na">useSslProtocol</span><span class="o">();</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"Error when enabling SSL"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="nc">String</span> <span class="nf">withConnection</span><span class="o">(</span><span class="nc">Function</span><span class="o"><</span><span class="nc">Connection</span><span class="o">,</span> <span class="nc">String</span><span class="o">></span> <span class="n">block</span><span class="o">)</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">(</span><span class="nc">Connection</span> <span class="n">connection</span> <span class="o">=</span> <span class="n">connectionFactory</span><span class="o">.</span><span class="na">newConnection</span><span class="o">())</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">block</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span><span class="n">connection</span><span class="o">);</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"Error during using rabbit connection"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
<span class="k">return</span> <span class="kc">null</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">accept</span><span class="o">(</span><span class="nc">String</span> <span class="n">payload</span><span class="o">)</span> <span class="o">{</span>
<span class="n">withConnection</span><span class="o">(</span><span class="n">c</span> <span class="o">-></span> <span class="o">{</span>
<span class="k">try</span><span class="o">(</span><span class="nc">Channel</span> <span class="n">channel</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">createChannel</span><span class="o">())</span> <span class="o">{</span>
<span class="n">channel</span><span class="o">.</span><span class="na">exchangeDeclare</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">exchangeName</span><span class="o">,</span> <span class="no">EXCHANGE_TYPE</span><span class="o">,</span> <span class="kc">true</span><span class="o">);</span>
<span class="n">channel</span><span class="o">.</span><span class="na">queueDeclare</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">queueName</span><span class="o">,</span> <span class="no">QUEUE_DURABLE</span><span class="o">,</span> <span class="no">QUEUE_EXCLUSIVE</span><span class="o">,</span> <span class="no">QUEUE_AUTODELETE</span><span class="o">,</span> <span class="kc">null</span><span class="o">);</span>
<span class="n">channel</span><span class="o">.</span><span class="na">queueBind</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">queueName</span><span class="o">,</span> <span class="n">config</span><span class="o">.</span><span class="na">exchangeName</span><span class="o">,</span> <span class="n">config</span><span class="o">.</span><span class="na">routingKey</span><span class="o">);</span>
<span class="n">channel</span><span class="o">.</span><span class="na">basicPublish</span><span class="o">(</span>
<span class="n">config</span><span class="o">.</span><span class="na">exchangeName</span><span class="o">,</span>
<span class="n">config</span><span class="o">.</span><span class="na">routingKey</span><span class="o">,</span>
<span class="kc">true</span><span class="o">,</span>
<span class="nc">MessageProperties</span><span class="o">.</span><span class="na">PERSISTENT_TEXT_PLAIN</span><span class="o">,</span>
<span class="n">payload</span><span class="o">.</span><span class="na">getBytes</span><span class="o">());</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"Creating rabbit channel failed"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="kc">null</span><span class="o">;</span>
<span class="o">});</span>
<span class="o">}</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">get</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">withConnection</span><span class="o">(</span><span class="n">c</span> <span class="o">-></span> <span class="o">{</span>
<span class="k">try</span><span class="o">(</span><span class="nc">Channel</span> <span class="n">channel</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="na">createChannel</span><span class="o">())</span> <span class="o">{</span>
<span class="n">channel</span><span class="o">.</span><span class="na">exchangeDeclare</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">exchangeName</span><span class="o">,</span> <span class="no">EXCHANGE_TYPE</span><span class="o">,</span> <span class="kc">true</span><span class="o">);</span>
<span class="n">channel</span><span class="o">.</span><span class="na">queueDeclare</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">queueName</span><span class="o">,</span> <span class="no">QUEUE_DURABLE</span><span class="o">,</span> <span class="no">QUEUE_EXCLUSIVE</span><span class="o">,</span> <span class="no">QUEUE_AUTODELETE</span><span class="o">,</span> <span class="kc">null</span><span class="o">);</span>
<span class="n">channel</span><span class="o">.</span><span class="na">queueBind</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">queueName</span><span class="o">,</span> <span class="n">config</span><span class="o">.</span><span class="na">exchangeName</span><span class="o">,</span> <span class="n">config</span><span class="o">.</span><span class="na">routingKey</span><span class="o">);</span>
<span class="nc">GetResponse</span> <span class="n">response</span> <span class="o">=</span> <span class="n">channel</span><span class="o">.</span><span class="na">basicGet</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">queueName</span><span class="o">,</span> <span class="kc">false</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">response</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">byte</span><span class="o">[]</span> <span class="n">body</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="na">getBody</span><span class="o">();</span>
<span class="kt">long</span> <span class="n">tag</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="na">getEnvelope</span><span class="o">().</span><span class="na">getDeliveryTag</span><span class="o">();</span>
<span class="n">channel</span><span class="o">.</span><span class="na">basicAck</span><span class="o">(</span><span class="n">tag</span><span class="o">,</span> <span class="kc">false</span><span class="o">);</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">String</span><span class="o">(</span><span class="n">body</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="kc">null</span><span class="o">;</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"Creating rabbit channel failed"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
<span class="k">return</span> <span class="kc">null</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">});</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>And finally the piece of code that checks that send() and receive() work as expected.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">class</span> <span class="nc">RabbitClientTests</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Logger</span> <span class="n">logger</span> <span class="o">=</span> <span class="nc">LoggerFactory</span><span class="o">.</span><span class="na">getLogger</span><span class="o">(</span><span class="nc">RabbitClientTests</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">RabbitClient</span><span class="o">.</span><span class="na">Config</span> <span class="n">config</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">RabbitClient</span><span class="o">.</span><span class="na">Config</span><span class="o">(</span>
<span class="s">"localhost"</span><span class="o">,</span> <span class="mi">5672</span><span class="o">,</span> <span class="s">"/"</span><span class="o">,</span> <span class="s">"guest"</span><span class="o">,</span> <span class="s">"guest"</span><span class="o">,</span> <span class="s">"q"</span><span class="o">,</span> <span class="s">"X"</span><span class="o">,</span> <span class="s">"key"</span><span class="o">,</span> <span class="kc">true</span><span class="o">);</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">ConnectionFactory</span> <span class="n">cf</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ConnectionFactory</span><span class="o">();</span>
<span class="kd">static</span> <span class="o">{</span>
<span class="n">cf</span><span class="o">.</span><span class="na">setUsername</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">username</span><span class="o">);</span>
<span class="n">cf</span><span class="o">.</span><span class="na">setPassword</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">password</span><span class="o">);</span>
<span class="n">cf</span><span class="o">.</span><span class="na">setHost</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">host</span><span class="o">);</span>
<span class="n">cf</span><span class="o">.</span><span class="na">setPort</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">port</span><span class="o">);</span>
<span class="n">cf</span><span class="o">.</span><span class="na">setVirtualHost</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">virtualHost</span><span class="o">);</span>
<span class="k">if</span> <span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">useSsl</span><span class="o">)</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="n">cf</span><span class="o">.</span><span class="na">useSslProtocol</span><span class="o">();</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">NoSuchAlgorithmException</span> <span class="o">|</span> <span class="nc">KeyManagementException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="n">logger</span><span class="o">.</span><span class="na">error</span><span class="o">(</span><span class="s">"Error when enabling SSL"</span><span class="o">,</span> <span class="n">e</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kd">final</span> <span class="nc">Broker</span> <span class="n">broker</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Broker</span><span class="o">();</span>
<span class="nd">@BeforeClass</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">beforeClass</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span>
<span class="kd">final</span> <span class="nc">BrokerOptions</span> <span class="n">brokerOptions</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">BrokerOptions</span><span class="o">();</span>
<span class="n">brokerOptions</span><span class="o">.</span><span class="na">setConfigurationStoreLocation</span><span class="o">(</span><span class="s">"qpid/broker-config.json"</span><span class="o">);</span>
<span class="n">broker</span><span class="o">.</span><span class="na">startup</span><span class="o">(</span><span class="n">brokerOptions</span><span class="o">);</span>
<span class="o">}</span>
<span class="nd">@AfterClass</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">afterClass</span><span class="o">()</span> <span class="o">{</span>
<span class="n">broker</span><span class="o">.</span><span class="na">shutdown</span><span class="o">();</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="nc">Channel</span> <span class="nf">getChannel</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span>
<span class="nc">Connection</span> <span class="n">connection</span> <span class="o">=</span> <span class="n">cf</span><span class="o">.</span><span class="na">newConnection</span><span class="o">();</span>
<span class="nc">Channel</span> <span class="n">channel</span> <span class="o">=</span> <span class="n">connection</span><span class="o">.</span><span class="na">createChannel</span><span class="o">();</span>
<span class="n">channel</span><span class="o">.</span><span class="na">exchangeDeclare</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">exchangeName</span><span class="o">,</span> <span class="s">"direct"</span><span class="o">,</span> <span class="kc">true</span><span class="o">);</span>
<span class="n">channel</span><span class="o">.</span><span class="na">queueDeclare</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">queueName</span><span class="o">,</span> <span class="kc">false</span><span class="o">,</span> <span class="kc">false</span><span class="o">,</span> <span class="kc">false</span><span class="o">,</span> <span class="kc">null</span><span class="o">);</span>
<span class="n">channel</span><span class="o">.</span><span class="na">queueBind</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">queueName</span><span class="o">,</span> <span class="n">config</span><span class="o">.</span><span class="na">exchangeName</span><span class="o">,</span> <span class="n">config</span><span class="o">.</span><span class="na">routingKey</span><span class="o">);</span>
<span class="k">return</span> <span class="n">channel</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">send</span><span class="o">(</span><span class="nc">String</span> <span class="n">message</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span>
<span class="nc">Channel</span> <span class="n">channel</span> <span class="o">=</span> <span class="n">getChannel</span><span class="o">();</span>
<span class="n">channel</span><span class="o">.</span><span class="na">basicPublish</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">exchangeName</span><span class="o">,</span> <span class="n">config</span><span class="o">.</span><span class="na">routingKey</span><span class="o">,</span> <span class="kc">null</span><span class="o">,</span> <span class="n">message</span><span class="o">.</span><span class="na">getBytes</span><span class="o">());</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kd">static</span> <span class="nc">String</span> <span class="nf">recv</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span>
<span class="nc">Channel</span> <span class="n">channel</span> <span class="o">=</span> <span class="n">getChannel</span><span class="o">();</span>
<span class="nc">GetResponse</span> <span class="n">response</span> <span class="o">=</span> <span class="n">channel</span><span class="o">.</span><span class="na">basicGet</span><span class="o">(</span><span class="n">config</span><span class="o">.</span><span class="na">queueName</span><span class="o">,</span> <span class="kc">true</span><span class="o">);</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">String</span><span class="o">(</span><span class="n">response</span><span class="o">.</span><span class="na">getBody</span><span class="o">());</span>
<span class="o">}</span>
<span class="nd">@Test</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">receive</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span>
<span class="nc">String</span> <span class="n">key</span> <span class="o">=</span> <span class="s">"{}"</span><span class="o">;</span>
<span class="nc">Supplier</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">supplier</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">RabbitClient</span><span class="o">(</span><span class="n">config</span><span class="o">);</span>
<span class="n">send</span><span class="o">(</span><span class="n">key</span><span class="o">);</span>
<span class="n">assertEquals</span><span class="o">(</span><span class="n">key</span><span class="o">,</span> <span class="n">supplier</span><span class="o">.</span><span class="na">get</span><span class="o">());</span>
<span class="o">}</span>
<span class="nd">@Test</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">send</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span>
<span class="nc">String</span> <span class="n">key</span> <span class="o">=</span> <span class="s">"{}"</span><span class="o">;</span>
<span class="nc">Consumer</span><span class="o"><</span><span class="nc">String</span><span class="o">></span> <span class="n">consumer</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">RabbitClient</span><span class="o">(</span><span class="n">config</span><span class="o">);</span>
<span class="n">consumer</span><span class="o">.</span><span class="na">accept</span><span class="o">(</span><span class="n">key</span><span class="o">);</span>
<span class="n">assertEquals</span><span class="o">(</span><span class="n">recv</span><span class="o">(),</span> <span class="n">key</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>This is how one can test RabbitMQ client. Clear and without much comments, as promised.</p>Sergey MelnychukWriting RabbitMQ client in Java is very well described in the tutorial and client docs, but writing test-coverage for such client can be tricky.