Friday, February 19, 2016

Documentation with code examples - the maintainable(?) way

We keep an internal base code platform at Nansen, which several of our sites are based. We like to think of this as our bread and butter. Meaning, those parts of every project you copy+paste to the next. Sadly, we have been quite bad at documenting it, or we have been documenting it, but the problem is often that the code examples tend to get out-of-date as code naturally evolve with time.

To alleviate this I wanted to create a way to keep the documentation and examples to support it up-to-date, but also contain readable text. One could argue that an SDK is a way, but SDK's usually contain way too much fluff in the way of tables upon tables of method, property and event definitions. And they are also plagued by the out-of-date example code problem above, as examples are written in the <example>-section of the summary documentation. It is easy to get lazy and forget about an example if a method or class is updated.

Anyways, my colleague Andreas 'Honungsprinsen' Oldeskog used a clever way to document our front-end framework by pulling markdown from github, and generating documentation pages from those files. I wanted to find a similar way, but with c# source code instead. This gave me three problems to solve:

  1. Having code examples that actually compile, and are relevant to the code I want to document.
  2. The ability to easily attach human readable documentation and informational text to said code.
  3. Some way of combining 1 and 2 in order to present it to the developers (hopefully) reading the documentation.

Now, for problem 1, part of the solution was so simple Andreas had to point it out to me: Use tests! I could just write unit tests, they run the actual code, and you can use assertions to verify the examples actually do what you show to the reader of the documentation.

The second part of the solution to problem 1 was finding a way to actually run the tests and present them to the reader. This was solved by using NUnit, and its test runner functionality somewhat hidden away in Nunit.core.dll, an assembly that is unfortunately not distributed along with nunit.framework.dll in the NUnit nuget package. I found it tucked away in a package named NUnit.Runners, and I had to manually reference the assemblies nunit.code.dll and nunit.core.interfaces.dll from my web project designated to run and present the results of my example tests.

When all that is achieved, it is pretty simple to actually run tests:

public class TestResultsEventListener : EventListener
{
    public TestResultsEventListener()
    {
        ListResults = new List<testresult>();
    }

    public List<testresult> ListResults { get; }

    public void TestFinished(TestResult result)
    {
        ListResults.Add(result);
    }

    // noop all other methods
}

public List<testresult> RunSomeTests(IEnumerable<string> testNames)
{
    var testPackage = new TestPackage("ExampleCode"); // this can be named anything
    testPackage.Assemblies.Add(Assembly.Load("my.assembly.with.tests").Location);

    var eventListener = new TestResultsEventListener();
    var runner = new SimpleTestRunner();
    if (runner.Load(_testPackage)) {
        var filter = TestFilter.Empty;
        if (testNames != null && testNames.Any()) {
            filter = new SimpleNameFilter(testNames.ToArray());
        }
        runner.Run(eventListener, filter, true, LoggingThreshold.All);
    }
    return eventListener.ListResults;
}

The result is a neat list of all our tests and their results, this solved problem 1, neat!
A bonus of this approach is also, since we use Episerver in our project platform, and the documentation/demo site is a basic implementation of said platform, all the service loading for epsierver is already done and the site is running so I can use real data for any code interacting with the site without having to mock my heart out (seriously, unit testing episerver code is 98% service mocking).

For problem 2, I decided to just use markdown syntax in the summary documentation of each test. If a more generic description or informative text was needed, I used the summary documentation of the test class. I also made sure to comment as much as possible of the test method code.

Now, problem 3 was trickier, I needed to somehow parse both the code and summary documentation for my test cases, and present the result. Cue Roslyn! Using the new Roslyn code parser Microsoft came up with, I could easily read a source file, and extract whatever I wanted.

Combining all three solutions above gave me a helper that:

  1. Found all tests for a specific class,
  2. Ran the tests, and recorded the results,
  3. Parsed the source code for said tests and formatted the result

The code for my helper:

public class ExampleCodeHelper
{
    private readonly DirectoryInfo _baseDirectory;

    private readonly TestPackage _testPackage;

    public ExampleCodeHelper(string baseDirectory, IEnumerable<string> assemblyNames)
    {
        _baseDirectory = new DirectoryInfo(baseDirectory);
        _testPackage = new TestPackage("ExampleCode");
        foreach (var assemblyName in assemblyNames) {
            _testPackage.Assemblies.Add(Assembly.Load(assemblyName).Location);
        }
        CoreExtensions.Host.InitializeService();
    }

    public static ExampleCodeHelper Default
    {
        get
        {
            var root = HttpContext.Current.Server.MapPath("~") + ConfigurationManager.AppSettings["examplecode.sourcefile.rootpath"];
            var assemblies = ConfigurationManager.AppSettings["examplecode.test.assemblies"].Split(';');
            return new ExampleCodeHelper(root, assemblies);
        }
    }

    private IEnumerable<string> CodeFiles => Directory.GetFiles(_baseDirectory.FullName, "*.cs", SearchOption.AllDirectories);

    private IEnumerable<TestResult> RunTests(IEnumerable<string> testNames)
    {
        var eventListener = new TestResultsEventListener();
        var runner = new SimpleTestRunner();
        if (runner.Load(_testPackage)) {
            runner.Run(eventListener, new SimpleNameFilter(testNames.ToArray()), true, LoggingThreshold.All);
        }
        return eventListener.ListResults;
    }

    public ClassDeclarationSyntax GetTestClass(string classFullName)
    {
        var definitions = GetAllTestClassDefinitions();
        var classSymbol = definitions.Keys.FirstOrDefault(k => k.ToString() == classFullName);
        return classSymbol != null ? definitions[classSymbol] : null;
    }

    /// <summary>
    /// Retrieves a <see cref="ExampleCodeContent"/> containing a number of <see cref="TestResultItem"/> objects with test results
    /// </summary>
    /// <param name="classFullName">Full name (including namespace) for the test class</param>
    public ExampleCodeContent GetTestResults(string classFullName)
    {
        var definitions = GetAllTestClassDefinitions();
        var classSymbol = definitions.Keys.FirstOrDefault(k => k.ToString() == classFullName);

        if (classSymbol == null) {
            return null;
        }

        var classDeclaration = definitions[classSymbol];

        var content = new ExampleCodeContent {
            Text = GetSyntaxNodeDocumentation(classSymbol),
            ContainerClass = classDeclaration
        };

        var testCases = classDeclaration.DescendantNodes()
                                        .OfType<MethodDeclarationSyntax>()
                                        .Where(m => HasNamedAttribute(m.AttributeLists, nameof(TestAttribute)))
                                        .ToList();

        var tests = RunTests(testCases.Select(t => $"{classSymbol.ToString()}.{t.Identifier.ValueText}"))
            .ToDictionary(key => testCases.Single(m => m.Identifier.ValueText == key.Test.MethodName), value => value);

        content.TestResults = tests.Keys.Select(method => new TestResultItem {
            Method = method,
            TestResult = tests[method],
            Text = GetSyntaxNodeDocumentation(GetDeclaredSymbol(method))
        }).ToList();

        return content;
    }

    /// <summary>
    /// Returns all test class definitions found in the defined source folder.
    /// </summary>
    /// <remarks>
    /// Only classes decorated with the <see cref="TestFixtureAttribute"/> attribute is returned.
    /// </remarks>
    public IDictionary<ISymbol, ClassDeclarationSyntax> GetAllTestClassDefinitions()
    {
        var classes = new List<ClassDeclarationSyntax>();
        foreach (var file in CodeFiles) {
            try {
                var testClasses = CSharpSyntaxTree.ParseText(File.ReadAllText(file))
                                                  .GetRoot()
                                                  .DescendantNodes()
                                                  .OfType<ClassDeclarationSyntax>()
                                                  .Where(c => HasNamedAttribute(c.AttributeLists, nameof(TestFixtureAttribute)));

                classes.AddRange(testClasses);
            }
            catch (Exception) {
                // just ignore failing source files
            }
        }

        return classes.ToDictionary(GetDeclaredSymbol, value => value);
    }

    #region helper methods

    private static bool HasNamedAttribute(SyntaxList<AttributeListSyntax> attributes, string attributeName)
    {
        var simpleAttributeName = attributeName.EndsWith("Attribute")
            ? attributeName.Substring(0, attributeName.LastIndexOf("Attribute", StringComparison.Ordinal))
            : attributeName;
        return attributes.SelectMany(a => a.Attributes).Any(a => a.Name.ToString().EndsWith(simpleAttributeName));
    }

    /// <summary>
    /// Retrives the <<paramref name="documentationSection"/>> documentation element "innertext" from the provided <see cref="ISymbol"/>
    /// </summary>
    private static string GetSyntaxNodeDocumentation(ISymbol symbol, string documentationSection = "summary")
    {
        var methodDocumentation = symbol.GetDocumentationCommentXml();
        if (!string.IsNullOrEmpty(methodDocumentation)) {
            var doc = XDocument.Parse(methodDocumentation);
            var nodes = doc.Descendants(documentationSection).FirstOrDefault()?.Nodes() ?? Enumerable.Empty<XNode>();
            return ParseSeeCrefs(string.Concat(nodes));
        }
        return null;
    }

    /// <summary>
    /// Changes any <see cref="TEXT" /> into <em>TEXT</em>, making it usable in html
    /// </summary>
    private static string ParseSeeCrefs(string inputText)
    {
        if (string.IsNullOrEmpty(inputText)) {
            return inputText;
        }
        var seeRegex = new Regex("< *see( +)cref=\"[^\"]:([^\"]+)\" */>");
        return seeRegex.Replace(inputText, m => "<em>" + m.Groups[2].Value + "</em>");
    }

    /// <summary>
    /// Returns the declared <see cref="ISymbol"/> for a specified <see cref="SyntaxNode"/>
    /// </summary>
    private static ISymbol GetDeclaredSymbol(SyntaxNode syntaxNode)
    {
        return CSharpCompilation.Create("MyCompilation", new[] {syntaxNode.SyntaxTree}).GetSemanticModel(syntaxNode.SyntaxTree).GetDeclaredSymbol(syntaxNode);
    }

    #endregion
}

With it, I can call ExampleCodeHelper.Default.GetTestResults("MyTestAssembly.MyTestResultClass") to compile a simple data model (source not shown), with documentation, and test source code. The result looking something like below:

Source code:

/// <summary>
/// #Content utilities#
/// <see cref="ContentUtility"/> contain a number of useful methods and extensions for Episerver content.
/// Basically, it can be seen as extension methods for <see cref="IContentLoader"/>. Though there are a number
/// of addiotional useful functions added.
/// 
/// Important to know is that the extensions, as opposed to the <see cref="IContentLoader"/>, will return null
/// instead of throwing a <see cref="EPiServerException"/> if not found or otherwise failing.
/// </summary>
[TestFixture(Category = "Episerver", Description = "Content utillties")]
public class ContentUtilityTests
{
    [Test]
    public void Load_Content()
    {
        var contentLink = SiteDefinition.Current.StartPage;

        //load content
        var content = contentLink.Get<IContent>();
        Assert.AreEqual(contentLink, content.ContentLink);

        //load content typed
        var startPage = contentLink.Get<PageData>();
        Assert.IsNotNull(startPage);

        // a wrong type or otherwise will return null
        var invalid = contentLink.Get<BlockData>();
        Assert.IsNull(invalid);
    }

    [Test]
    public void Load_content_using_a_specific_language()
    {
        var contentLink = SiteDefinition.Current.StartPage;
        var content = contentLink.Get<PageData>(LanguageLoaderOption.FallbackWithMaster());
        Assert.IsNotNull(content);
    }
}

Rendered result:


27 comments :

  1. It’s hard to point to a single, cool feature of the Stripe API documentation. In truth, there aren’t any shiny bells and whistles — and that’s probably what makes the Stripe API reference so good.
    ebook publishing

    ReplyDelete
  2. There is a strong argument that it is not safe for payforessay log in companies writing essays to be used by students from different institutions, they are good at helping everywhere.

    ReplyDelete
  3. We stumbled over here by a different website and thought I might check things out. I like what I see so now i am following you 룰렛

    ReplyDelete
  4. Such an amazing and helpful post. I really really love it.
    온라인카지노

    ReplyDelete
  5. This was an extremely nice post. Taking a few minutes and actual effort to generate a top notch article.
    카지노사이트

    ReplyDelete
  6. it’s awesome and I found this one informative
    스포츠토토

    ReplyDelete
  7. We are really grateful for your blog post for giving a lot of information
    카지노사이트

    ReplyDelete
  8. Are you the one who studies this subject?? I have a headache with this subject.우리카지노Looking at your writing was very helpful.

    ReplyDelete
  9. 바카라사이트Dec 14, 2021, 9:59:00 AM

    Looking at this article, I miss the time when I didn't wear a mask. 바카라사이트 Hopefully this corona will end soon. My blog is a blog that mainly posts pictures of daily life before Corona and landscapes at that time. If you want to remember that time again, please visit us.


    ReplyDelete
  10. I am a 오공슬롯 expert. I've read a lot of articles, but I'm the first person to understand as well as you. I leave a post for the first time. It's great!!

    ReplyDelete
  11. MOST RIDICULOUS & UNBELIEVABLE 메이저공원 Morning JACKPOTS EVER!

    ReplyDelete
  12. My curiosity was solved by looking at your writing. Your writing was helpful to me. 룰렛사이트 I want to help you too.

    ReplyDelete
  13. This is the perfect post.메이저토토사이트 It helped me a lot. If you have time, I hope you come to my site and share your opinions. Have a nice day.

    ReplyDelete
  14. Thank you The international travelers who travel to Kenya need e visa to Kenya. That they can apply online and can get the 24*7 customer assistant.

    ReplyDelete
  15. That's a great article! The neatly organized content is good to see. Can I quote a blog and write it on my blog? My blog has a variety of communities including these articles. Would you like to visit me later? 토토사이트추천

    ReplyDelete
  16. I had a lot of fun at this Olympics, but something was missing. I hope there's an audience next time.안전토토사이트

    ReplyDelete
  17. The website is good and the stock is good. Thanks for all you do. India urgent visa, you can fill an online Indian emergency visa application form. Within 5 to 10 minutes you can fill your India emergency visa application form. Urgent visa to India you get in 1 to 3 business days.

    ReplyDelete
  18. I think this is an informative post and it is very useful and informative. So, I would like to thank you for the effort you put into writing this article. Apply e visa Indian online & check Indian e visa photo requirements via eta Indian e visa website.

    ReplyDelete
  19. Captivating post. I Have Been contemplating about this issue, so an obligation of appreciation is all together to post. Completely cool post.It 's greatly extraordinarily OK and Useful post.Thanks 사설토토사이트

    ReplyDelete
  20. What a great explanation in yours posts.. International travelers who wish to travel to Azerbaijan for tourism and business purpose need to apply for Azerbaijan electronic visa through e visa application.

    ReplyDelete
  21. This article presents clear idea designed for the new visitors of blogging, that in fact how to do blogging and site-building. 경마

    ReplyDelete
  22. you have done a great job. I will definitely dig it and personally recommend to my friends. 슬롯머신

    ReplyDelete
  23. Thank you for sharing with us, I conceive this website genuinely stands out 토토사이트

    ReplyDelete
  24. Register with our website here, you won't be disappointed. Because we are known for giving away unlimited free credits. betflix

    ReplyDelete
  25. You've just gotten an irate developer's email. Something is amiss with your documentation, and it took the developer hours to figure it out. It's now up to you to update the documentation and find out how to avoid such problems in the future. But how? It's difficult to produce excellent documentation. Working on it typically necessitates disregarding another aspect of your career, but that time might be just as vital as your development efforts. A few hours per week spent enhancing documentation may add up quickly. Developers will be less likely to get stuck, resulting in fewer help requests and, ideally, fewer furious emails. In fact, having excellent developer documentation may result in pleasant, gushing emails. You should Visit Nerd's Magazine for more info.

    ReplyDelete