libocon-2023-workshop/bugzilla.md

LibreOffice Quality Assurance with Bugzilla

Welcome to this session!

We are going to learn about Quality Assurance (QA) in the LibreOffice project, using Bugzilla as our preferred issue-tracking system.

This session is supposed to be hands-on, so we can all get some experience in triaging bug reports to better understand the process – and straight away contribute something to the project!

Remember that there are no silly questions. If anything doesn't make sense, or if you are curious about something, please don't hesitate to interrupt.

Bugzilla

The Document Foundation uses the Bugzilla software to track issues and enhancement requests from reporting to their resolution. It is mainly used for LibreOffice, but other projects are on there too, like the Document Liberation Project or the LibreOffice Impress Remote app. Our instance can be found at bugs.documentfoundation.org.

On Bugzilla, anyone can create and account and report an issue or suggest an enhancement. QA contributors will then "triage" the reports, which might involve any of the following:

• Figure out if the issue is reproducible
• Has it been already reported?
• Update the metadata, categorise it
• Copy relevent people in
• Draw links to other reports
• Ask for more information
• Close the report if it is not a bug, or a bug unrelated to LibreOffice
• Weed out spam
• ... and more.

There is a wealth of activities to choose from, and because LibreOffice is such a large project, there is enough to never get bored!

Bug reporting

Let's start with looking at what information makes a bug report useful and easy to triage. Let's do that by testing how a bug report is created.

There are two different forms to report LibreOffice bugs, which you can bookmark:

• A simpler, more guided way (better for beginners)
• A more advanced one (with more fields, and an extra useful quick search functionality)

Let's have a look at each of of them to familiarise ourselves with their differences and the various fields available.

Guided form

The guided form gives more information around a smaller number of fields, to guide the contributor in creating a LibreOffice bug report that has clear steps, version information, and expected results. You can fill in the following:

• Summary: the title of the report, which should summarise but contain enough information to differentiate from other reports
• Description: general description of the issue
• Component: which part of LibreOffice is affected? Note that sometimes, you just have to pick one of several possible values (e.g. an issue that affects both Draw and Impress)
• Hardware: architecture of your device
• Operating system: your OS. You can use "all" if you have tested several.
• Version: which is the earliest version affected? (As far a you can tell!)
• Reproducibility: how often does the issue occur when testing?
• Severity: how bad is it?
• Steps to reproduce: the simplest steps you can provide to trigger the issue. Can be from a new session, or starting with a document you can later attach.
• Actual results: what is the resulting issue?
• Expected results: what you expect to happen instead
• Other information: anything else that might help? Paste here the version information, which you can easily copy with the button in Help > About LibreOffice (or LibreOffice > About on macOS)
• User profile reset: various issues can be resolved by using a new user profile, so remember to try that (or restart in safe mode).

Advanced form

The advanced form gives you direct access to more fields and functionalities, expecting you to be a more experienced QA contributor. (But these fields are available anyway after the bug was created in the guided way.)

• Importance: these two fields are a bit trickier, and some values can only be used by a certain group of contributors. If you are unsure about this one, leave it with its defaults! An example flow chart (ODG, PNG) is however available to pick a good value.
  • Severity: how bad the bug is. For example, most crashes should be "Critical". If you are requesting a new feature, set it to "Enhancement".
  • Priority: how prioritised the bug should be. For example, if an issue is only a small cosmetic annoyance and affects few people, set it to "trivial".
• Assignee: only use this to assign yourself if you plan to submit a patch to fix it.
• CC: any other contributor who might be interested (or responsible for a regression)
• Alias: ignore this one, as it is mainly used for meta bugs (more on that below).
• URL: is there a forum post or ask.LO topic linked to this issue?
• Crash report or signature: ID or signature of an associated crash report (the one you might get with the Crash Report dialog)
• Attachment: attach any file (a document to reproduce the issue, a screenshot, a backtrace...), anything that helps understand the issue. But keep in mind that explanations and version information are better contained in the bug description and comments!
• Keywords: a fixed keyword that categorises the bug report (see the Keywords section below)
• Depends: does this issue depend on another issue to be fixed before it can itself be fixed? (For various values, separate them with a comma.)
• Blocks: does this issue block the resolution of another issue? This helps creating a relationship chain, and is mostly used for meta bugs – reports used to categorise other reports. More on this in the [meta bugs]([Meta bugs](#Meta bugs)) section below! (For various values, separate them with a comma.)
• See also: URLs linking to other bug reports that are related (separate the URLs with a space)

More fields

After creating a report, you will see extra fields:

• Whiteboard: a more free form (but not entirely free!) field for more information. This is sometimes automatically populated, for example when an associated bug fix is commited. More on this in the Whiteboard section.
• Personal tags: use this for your own sorting of reports. Changing this field will not notify anyone.

You can find more information on Bugzilla fields on our wiki.

Other products

If you want to file a bug against another project, like Impress Remote, Document Liberation Project, or the QA Tools, start on the bug report start page.

What makes a good bug report?

To help the QA team and developers understand, triage and fix issues, here are some of the most important points:

• Test with a recent version. Maybe the issue is already resolved? Mention all versions you tested in a comment.
• Do a few searches with different keywords: has the issue been reported already? If that's the case, don't report it again!
• Clear and to the point. No need to go into details if the details are not needed to reproduce or understand the issue.
• Steps that are easy to follow. As short and simple as possible. If starting from scratch is too complex, provide an example file that makes it easier.
• Actual vs Expected results. Remember that different users might have different expectations, so be clear about what you believe the issue is exactly.
• Full version information copied and pasted from the About dialog.
• Be nice! A lot of contributors do it on their spare time, to help out. Please be curteous, polite. Communicating by text can often be misinterpreted, so communicate clearly and assume the best from others. And in any case, we have a Code of Conduct we ask everyone to follow.

More details on fields

What about these "keywords" and this "whiteboard"? How are they used?

Keywords

Keywords are hard-coded: their description is available on the wiki.

They are often used to categorise:

• which import or export filter is affected, e.g. "filter:svg"
• if the issue is an "accessibility" issue
• if the issue is a "regression": something that used to work well before but does not anymore. Related to that are "bibisectRequest", "bibisected" and "bisected".
• if the issue is an "implementationError": something that does not work since it was implemented, for example part of a new feature
• if the issue involved loss of data with "dataLoss"
• if the issue requires the opinion of the UX/Design team, using the "needsUXEval". (And don't forget to add libreoffice-ux-advise in the CC as well!)
• "easyHack" issues, to point new developers to bugs that they can focus on to start with
• the need for or presence of a backtrace: "needBacktrace" and "haveBacktrace"

These values are comma-separated.

Whiteboard

The Whiteboard field is more free-form, but we do have a list of commonly-used values. For example:

• Automatically added:
  • "target" values, letting contributors know in which version the bug fix will be available, e.g. "target:24.0"
  • "QA:needsComment" for when no one other than the reporter has commented yet.
• Manually added:
  • to mention that a new feature is available in the corresponding major release notes, you can mark it as such, e.g. "inReleaseNotes:24.0"

These values are separated by spaces.

Meta bugs

Meta bugs have [Meta] in their summary and are used to categorise other reports. They are useful to mark reports as affecting different topics and aspects of the software, and offer more precise categorisation than Component or Keyword values. Contributors can then subscribe to meta bugs of interest so they get notified when there is movement.

For example, if an issue relates to the Orca screen reader unable to pick up some information in the Special Character dialog, you can add 36549 and 109232 in its "Blocks" field.

You can find a handy searchable list of meta bugs on the wiki – pin this one when triaging!

Searching for bugs

The simple search on Bugzilla will match keywords in many places in the report, and will return only open bug reports, which is great for a quick search.

However, the advanced search allows creating more complex queries, for example: a regression that was fixed for version 7.6 and has "spelling" in its summary.

• Summary: "spelling"
• Keyword: "regression"
• Whiteboard: "target:7.6"
• Status: "Resolved/Verified - FIXED"

Tip: make sure you try variations on keywords, like US vs UK spellings, or parts of words! For example:

• search for both "colour" and "color" if that term is relevant
• search for "track" instead of "tracking" or "tracked" to catch more results

The results can then be navigated, the search can be refined with "Edit search", and your favourite queries can be saved to have a quick link at the bottom of the page.

Triaging and the Status field

When an issue is first reported, it starts its life with the "Unconfirmed" status.

"Triaging" refers to processing bug reports that are "unconfirmed".

The main task is to confirm that the bug is indeed present to then mark it as "new", but a thorough triaging might involve any of the following:

• searching if the issue has been reported before
• checking if the issue is a regression by testing older versions
• checking if the fields have been populated properly, or if values are missing
• asking for more information to the bug reporter - setting the status to "NEEDINFO"
• tweaking the Priority and Severity
• linking to relevant other reports in "See Also"
• categorising with relevant Meta bugs in "Blocks"

Closing the report

If a bug is confirmed, it will usually require a fix before is can be marked as "Resolved - FIXED" (which is done by the bug fixer).

However, triaging often leads to closing a report for a variety of reasons, e.g.:

• The issue is already reported elsewhere: Resolved - DUPLICATE
• Actually not a bug, the software works as expected (in which case you can point to the documentation): Resolved - NOTABUG
• The issue is resolved without knowing exactly what fixed it (for example, after a profile reset): Resolved - WORKSFORME
  • If the issue was resolved in a specific version, it is useful to identify what fixed it, to eventually mark as a duplicate or as "Resolved - FIXED".
• The report does not make sense: Resolved - INVALID
• The issue lacks essential information, and it was not provided after having requested it: Resolved - INSUFFICIENTDATA
• The issue is not in LibreOffice, but rather pertains to another app, the OS or the Desktop Environment: Resolved - NOTOURBUG
• The issue has been moved to another tracker where it belongs (for example, a website bug moved to Redmine): Resolved - MOVED
• After consulting with developers and/or the UX/Design team, if there is an issue but it is decided that it won't be fixed: Resolved - WONTFIX

Regressions

The importance of identifying regressions

"Regressions" are issues that arose in a version but weren't present in previous ones. For example, something that used to work well but is now broken. Note that this is not to be confused with an "implementation error": a new feature that does not work as planned.

It is important to identify regressions (as opposed to issues that are inherited from OpenOffice.org) so the developer who introduced it can quickly fix it.

It is also important to triage regressions quickly, so it is easier to fix and affects less users.

So how do we identify them, and then figure out exactly when it started?

1. If the issue is reproduced, test it in older versions. You can have several versions of LibreOffice installed in parallel to achieve that, including daily master builds. If the problem didn't exist in earlier vesions, report the version that works, the version that doesn't, and use the Keywords "regression, bibisectRequest".
2. If you are able to, bibisect the issue to pinpoint at which commit the issue started. Learn more in the next section! If not, other contributors will search for "bibisectRequest" issues and take care of it.

Bibisecting regressions

Bibisection is short for "binary bisection", a process used to pinpoint exactly when a problem (or a fix) appeared.

The idea is to slice a large number of changes into two, check if the issue is present or not at the midpoint, and continue the process with the relevant half, until you get to the commit that started it all.

This is a more advanced process that involves a little bit of command line knowledge, but our wiki explains the process in detail. To bibisect something, you will need:

• Git
• A bit of disk space for the relevant repository (up to 10 GB)
• A bit of time (but it gets quicker as you get used to it!)

Note that it is not used only for regressions: if a bug was fixed in a specific version, but we are not sure what fixed it (i.e. resolved as "works for me"), we can bibisect what and who fixed the bug, so we can give them credit and have better QA data.

You can try bibisecting in your own time with one of the bugs needing it, and if you need assistance, please reach out on the chat or via email!

Let's practice!

The best way to learn is to get our hands dirty and start looking through some issues.

To make sure we don't step on each other's toes, use the list of lists of bugs on our collaborative pad to find unconfirmed bug reports. Please add your name in front of the link so others know you are working on it!

Once you have your list, find a report that sounds interesting and apply what we learned above. Here are some suggested questions to ask yourself:

• Has this been reported before? If so, mark as a duplicate!
• Can I reproduce in a recent release? In a master build? Make sure to include your version info in the comment!
• Is it actually a bug or is it normal behaviour?
• Is it a regression?
• Can I categorise it with a meta bug number in the "Blocks" field?
• Is there anything to fix in the metadata?

When you edit a report, make sure you add yourself to the CC so you can follow-up and see how others will contribute to it!

More information

There's always more to learn. The following sections are not essential to get started, but let's talk about them as they become relevant.

Testing further

Pinpointing an issue more precisely often involves the following:

New user profile

To rule out extensions, user configuration and corrupted user profiles, one can test in Safe Mode, backup their current user profile, or completely delete it.

More on that on the User Profile page on the wiki.

Skia

Skia, a graphics library, can often be the culprit in graphical issues and crashes. If "Skia" appears in a bug reporter's version information, it is useful to ask them to test with forced software rendering ("Skia/Raster") or turned entirely off.

More on that in the Skia section on the wiki.

VCL plugins

Visual Class Library plugins – or VCL plugins – are used to adapt the UI to various widget toolkits, for example for Linux distributions using KDE Frameworks, GTK or QT.

You can launch LibreOffice from the command line using a VCL plugin different to you default one, using an environment variable. For example, to use the KF5 plugin:

SAL_USE_VCLPLUGIN=kf5 libreoffice7.6

Or the "generic" VCL plugin:

SAL_USE_VCLPLUGIN=gen libreoffice7.6

More on that on the Environment Variables page on the wiki.

Verifying fixes

Part of QA is to also verify that something marked as "fixed" is actually fixed and works as expected.

If you want to, you can look for fixed bugs or enhancements, test a version of LibreOffice that integrates that change, and confirm that it is indeed working – or report follow-up issues if they exist!

Once verified, you can mark the report as "Verified - FIXED" – and thank the developer! 🙏

Automatic linking

To make navigation easier, Bugzilla automatically creates links when particular strings are detected in comments:

• "tdf#12345" or "bug 12345" links to the relevant report
• "comment 13" links to the comment on the page
• "bug 12345 comment 13" links to the relevant comment in another report
• "attachment 12345" links to the relevant attachment (even from another bug report, useful to reuse test files)
• a commit SHA links to the commit's details

You can use the Preview to see if the links work as expected.

Pre-written messages

As triaging can get a bit repetitive, we have a handy list of pre-written responses that you can make use of. But remember that personalising a message or writing a new one is often nicer than sounding like a robot! 😄

Comment tags

Comment tags can be used to categorise comments, and often clarify the conversation:

• "obsolete" if not relevant anymore
• "off-topic" to hide side-conversations or unhelpful information
• "spam" for... spam. (See Spam below.)
• "me-too" for no-value comments that only mention the commenter is also affected without adding relevant information like the version tested

See other possible comment tag values.

Spam

Bugzilla is unfortunately the target of nasty spammers on a frequent basis.

When you see a new report created for spamming:

1. Switch the Component to "Deletion Request"
2. Close the report as "Resolved - Invalid"
3. Remove e.g. spam URLs from the fields
4. Mark any spam comment as "spam"

If the spam is added to an existing, valid report:

1. Mark the spam comment as "spam"
2. Remove spam URLs, fix bogus changes to fields
3. If the report had already been resolved previously for valid reasons, change it from "Resolved" to "Closed" as this status will block new comments.

In any case, let others in the chat know about it so repeat offenders can be blocked.

Automated tasks

Bugzilla does a bit of automatic work to make it easier for contributors. For example, if a bug has not been touched for a while, a comment will be added to ask to test the issue again in a currently-supported version of LibreOffice.

Note that the use of the NEEDINFO status is particular: if a bug report has this status for 6 months without a response, it is automatically closed as "INSUFFICIENTDATA". This means that it is preferable to not use it if the bug has been confirmed (i.e. set as "NEW") previously. As a rule of thumb, only use this status for unconfirmed reports.

Join the team!

We are always looking for help in the QA space, so please join the team! Come have a chat, look at the wiki and get triaging some more bugs to learn new skills.

Here are important links and resources to get you started:

• QA on the wiki: the home page of QA activities
• Communication:
  • Chat with the team: IRC, which is bridged to Telegram (Matrix bridge is currently down). A great place to get notified of new reports and ask questions about QA.
  • Subscribe to the (low volume) mailing list for questions and annoucements
• Read the QA Blog
• Bug reporting and triage:
  • Basics of reporting a bug
  • Triaging reports
  • List of unconfirmed bugs to triage
• Bibisecting:
  • Bibisect on the wiki
  • List of regressions to bibisect
• QA Dashboard for statistics and visualisations

Licence

This material is licensed under a Creative Commons Attribution license (CC-BY 4.0). You are therefore free to:

• Share — copy and redistribute the material in any medium or format
• Adapt — remix, transform, and build upon the material

... as long as you give attribution, i.e. you give appropriate credit to the original author, and link to the license.