Finn Upham

• Upham, Finn; H?ffding, Simon & Rosas, Fernando E. (2024). The Stilling Response: From Musical Silence to Audience Stillness. Music & Science. ISSN 2059-2043. 7, p. 1¨C14. doi: 10.1177/20592043241233422.
• Upham, Finn; Lee, Jin Ha & Park, So Yeon (2023). Audience Reconstructed: Social media interaction by BTS fans during livestream concerts. Frontiers in Psychology. ISSN 1664-1078. 15. doi: 10.3389/fpsyg.2024.1214930. Show summary

  COVID-19-motivated social distancing made online concerts common practice in 2020 and 2021, with millions logging into streaming sites to see their favorite artists perform in realtime. For some fans, watching alone at home may have been enough, but concert-concurrent surges of social media activity suggest many virtual performance attendees are doing more. To understand why fans would turn their attention from these precious performance streams to social media, we explored Twitter engagement during four live streamed concerts performed by the Kpop group BTS in 2021. In public Tweets sampled by either concert hashtag or a predefined stream of users and keywords, we evaluated patterns in posting rates in relation to concert program events and investigated the content patterns in 1,200 Tweets sampled from four ranges of popularity (number of Retweets during the concert). Across concerts, short "Shout" Tweets surged at the start of songs, while the rate of retweets often fell during musical performances and shot up when BTS was off stage. Content analysis on the subsample found the materials most widely shared were informational or featured concert visuals, mimicking how fans use their phones at in-person concerts. Most original posts received few Retweets and were more personal and expressive of admiration for the performers. Comparison between the samples (concert hashtag vs. stream) also suggests users were strategic in using or omitting official concert hashtags with the strongest differences in the most widely disseminated content. Postings on Twitter during these performances seemed principally directed to fellow fans and audience members, by individuals choosing to share their own excitement and check in with others. By leveraging their existing social media networks, these concert attendees constructed a collective and interactive concert space, connecting with friends and strangers in the crowd and helping each other capture a richer experience than any broadcasting platform currently supports.

• Riaz, Maham; Upham, Finn; Burnim, Kayla; Bishop, Laura & Jensenius, Alexander Refsum (2023). Comparing inertial motion sensors for capturing human micromotion, Proceedings of the Sound and Music Computing Conference 2023. SMC Network . ISSN 978-91-527-7372-7. doi: 10.5281/zenodo.8316051. Full text in Research Archive Show summary

  The paper presents a study of the noise level of accelerometer data from a mobile phone compared to three commercially available IMU-based devices (AX3, Equivital, and Movesense) and a marker-based infrared motion capture system (Qualisys). The sensors are compared in static positions and for measuring human micromotion, with larger motion sequences as reference. The measurements show that all but one of the IMU-based devices capture motion with an accuracy and precision that is far below human micromotion. However, their data and representations differ, so care should be taken when comparing data between devices.

• Swarbrick, Dana; Upham, Finn; Erdem, Cagri; Jensenius, Alexander Refsum & Vuoskoski, Jonna Katariina (2022). Measuring Virtual Audiences with The MusicLab App: Proof of Concept. In Michon, Romain; Pottier, Laurent & Orlarey, Yann (Ed.), Proceedings of the 19th Sound and Music Computing Conference. SMC Network. ISSN 9782958412609. p. 532¨C539. doi: 10.5281/zenodo.6798290. Full text in Research Archive Show summary

  We present a proof of concept by using the mobile application MusicLab to measure motion during a livestreamed concert and examining its relation to musical features. With the MusicLab App, participants¡¯ own smartphones¡¯ inertial measurement unit (IMU) sensors can be leveraged to record their motion and their subjective experiences collected through survey responses. The MusicLab Lockdown Rave was an Algorave (live-coded dance music) livestreamed concert featuring prolific performers Renick Bell and Khoparzi. They livestreamed for an international audience who wore their smartphones with the MusicLab App while they listened/danced to the performances. From their acceleration, we computed quantity of motion and compared it to musical features that have previously been associated with music-related motion, namely pulse clarity and low and high spectral flux. By encountering challenges and implementing improvements, the MusicLab App has become a useful tool for researching music-related motion.

View all works in Cristin

• Lartillot, Olivier; Swarbrick, Dana; Upham, Finn & Cancino-Chac¨®n, Carlos Eduardo (2023). Video visualization of a string quartet performance of a Bach Fugue: Design and subjective evaluation.
• Upham, Finn (2023). Making the best of it: Constraints and opportunities in studying complex human musicality.
• Upham, Finn (2023). Insight into human respiration through the study of orchestras and audiences.
• Upham, Finn & Christophersen, Bj?rn Morten (2023). Bodies in Concert: RITMO project with the Stavanger symfoniorkester.
• Upham, Finn & Oddekalv, Kjell Andreas (2023). Fingers and Tongues: Appreciating Rap Flows through Proprioceptive Interaction in Rhythm Hive. Show summary

  Rhythm games have been studied for their potential to develop interest in music making (Cassidy and Paisley, 2013) and transferable musicianship skills (Richardson and Kim, 2011), but how might they influence players appreciation for specific musical works? Proprioceptive interaction, a concept by game designer Matt Bloch (Miller, 2017), refers to changes in a game player's perception of music as they practice specific movements to it. By drawing attention to coincidental sounds, players can develop their hearing and appreciation for nuances of production and performance. Many fans of rap enjoy performances in languages they do not speak themselves. Without specific language skills, expertise in rap performance, and/or time to learn lyrics phonetically, their experience of a rap flow is hampered by an inability to imitate and imagine the generative action of performance. Rhythm Hive is a mobile rhythm game based on the music of BTS, Enhyphen, and TXT, Kpop groups with substantial followings outside of Korea. Game play presents players with finger choreographies to these groups¡¯ hit songs, tapping sequences to the vocal performances across four to seven positions in a line. For these groups¡¯ many non-rapping and non- Korean-speaking fans, playing Rhythm Hive may offer deeper understanding of performances by rappers like RM, Suga, and J-Hope. Through expert analysis of rap performance, transcriptions of game play, and reflections on the experience of playing Rhythm Hive, we consider shared structure between the prescribed finger choreographies and the rap flows they accompany. We studied rap verses from four BTS songs along side their Easy and Hard level tapping sequences (vocal versions only) to identify parallels in rhythm, segmentation, repetition, and accents. Easy mode choreographies tend to mark their relationship to rap vocals by hitting the start of lines and then articulating structure with repeated contours tapped on quarter and eighth notes. Hard mode choreographies tend to hit every rapped syllable and incorporate more gestural flourishes to mark pitch changes, ending and internal rhymes, and interesting breaks from a steady 16th note flow. Both Easy and Hard tappings sequences consistently follow the rap track when it deviates from a quantized beat. The finger choreographies of Rhythm Hive illuminate rap performances by directing and rewarding players¡¯ attention to details of flows that may otherwise be missed. Game feedback pushes players to replicate delivery microtiming, while spatial patterns underline linguistic and rhythmic structure. Hard mode tapping sequences articulate distinguishing characteristics of specific rap styles, given players tangible sensitivity to degrees of technicality and nuances of genre. While fans may be motivated to play rhythm games like Rhythm Hive out of a preexisting love of the music and bands, tapping along offers them a chance to attend to, appreciate, and even rehearse key aspects of these rappers¡¯ expert performance choices, regardless of how well they might follow by ear.

• Upham, Finn (2023). Using Metrically-entrained Tapping to Align Mobile phone sensor measurements from In-person and Livestream Concert Attendees. Show summary

  Music is often made and enjoyed in large groups, but simultaneously capturing measurements from dozens or hundreds of people is technically difficult. When measurements are not constrained to wired or continuous connected wireless systems, we can record much bigger groups, potentially taking advantage of the wearable sensors in our phones, watches, and more dedicated devices. However, aligning measurements captured by independent devices is not always possible, particularly to a precision relevant for music research. Phone clocks differ and update sporadically, wearable device clocks drift, and for online broadcast performances, exposure times can vary by tens of seconds across the remote audience. Many measurement devices that are not open to digital synchronisation triggers still include accelerometers; with a suitable protocol, participant movement can be used to imbed synchronisation cues in accelerometry measurements for alignment regardless of clock times. In this paper, we present a tapping synchronisation protocol that has been used to align measurements from phones worn by audience members and a variety sensors worn by a symphony orchestra. Alignment with the embedded cues demonstrate the necessity of such a protocol, correcting offsets of more than 700 ms for devices supposedly initialised with the same computer clock, and over 10 s for online audience participants. Audience tapping performance improved cell phone measurement alignment to a median of 100 ms offset, and professional musicians tappings improved alignment precision to around 40 ms. While the temporal precision achieved with entrained tapping is not quite good enough for some types of analyses, this improvement over uncorrected measurements opens a new range of group coordination measurement and analysis options.

• Upham, Finn (2023). Breathing Together in Music, a RESPY Workshop. Show summary

  Respiration is a subtle but inescapable element of real time musical experiences, sometimes casually accompanying whatever we are hearing, other times directly involved in the actions of sound generation. This workshop explores respiratory coordination in music listeners and ensemble musicians with respy, a new python library for evaluating respiration information from single belt chest stretch recordings. Following an introduction to the human respiratory system and breathing in music, the workshop demonstrates how the respy algorithms evaluate phase and breath type, and presents statistical tools for assessing shared information in these features of people listening to or making music together. Rather than only use aggregate statistics such as respiration rate, respy aims to elevate the details of the respiratory sequence to facilitate our exploration of how breathing is involved in musical experiences, second-by-second. Measurable coordination of the respiratory system to musical activities challenges our expectations for interacting oscillatory systems. This session will conclude with a discussion on the different categories of relationships possible between people breathing together in music.

• Bishop, Laura & Upham, Finn (2023). Bodies in Concert. Show summary

  Increasingly, research on music performance is moving out of controlled laboratory settings and into concert halls, where there are opportunities to explore how performance unfolds in high-arousal conditions and how performers and audiences interact. In this session, we will present findings from a series of live research concerts that we carried out with the Stavanger Symphony Orchestra. The orchestra performed the same program of classical repertoire for four audiences of schoolchildren and an audience of families. Orchestra members wore sensors that collected cardiac activity, respiration, and body motion data, and the conductor additionally wore a full-body motion capture suit and eye-tracking glasses. Audience members in some of the concerts were invited to wear reflective wristbands, and wristband motion was captured using infrared video recording. We will begin the session with a discussion of the scientific and methodological challenges that arose during the project, in particular relating to the large scale of data capture (>50 musicians and hundreds of audience members), the visible nature of research that is carried out on a concert stage, and the development of procedures for aligning data from different recording modalities. Next, we will present findings from two lines of analysis that investigate different aspects of behavioural and physiological coordination within the orchestra. One analysis investigates the effects of audience noise and musical roles on coherence in (i) cardiac rate and variability and (ii) respiratory phase and rate. The second analysis investigates the effects of musical demands on synchronization of body sway, bowing, and respiration in string sections. We will conclude the session with an open discussion of how live concert research might be optimized.

• Martin, Remy Richard; Cross, Ian; Upham, Finn; Bishop, Laura; S?rb?, Solveig & ?land, Frederik (2023). What can one learn from more naturalistic concert research?
• Riaz, Maham; Upham, Finn; Burnim, Kayla; Bishop, Laura & Jensenius, Alexander Refsum (2023). Comparing inertial motion sensors for capturing human micromotion. Show summary

  The paper presents a study of the noise level of accelerometer data from a mobile phone compared to three commercially available IMU-based devices (AX3, Equivital, and Movesense) and a marker-based infrared motion capture system (Qualisys). The sensors are compared in static positions and for measuring human micromotion, with larger motion sequences as reference. The measurements show that all but one of the IMU-based devices capture motion with an accuracy and precision that is far below human micromotion. However, their data and representations differ, so care should be taken when comparing data between devices.

• Upham, Finn (2022). Respiratory phase alignment in music audiences.
• Upham, Finn (2022). Uncovering the active listener. Show summary

  Our experiences of music are both highly idiosyncratic, special to each of us in each moment, and collective, with common influences across a listening crowd. Through a series of empirical studies on how people feel and behave during music listening, Finn Upham traces their trajectory from a basic model of performed stimulus and audience response to an empowered-listener view of musical engagement. If participation is part of all musical experiences, this poses a question to all producers of music: ¡°What are you asking your audience to do?¡±

• Upham, Finn; Memis, Ahmet Emin; Hansen, Niels Chr.; Rosas, Fernando E.; Clim, Maria-Alena & Jensenius, Alexander Refsum (2022). Participatory applause: Interactions of audience members clapping at the end of a classical music concert. Show summary

  Participatory applause: Interactions of audience members clapping at the end of a classical music concert According to musicological studies of audience culture, applause is the most overt form of participation allowed to the collections of individuals attending classical music concerts (Brandl-Risi, 2011; Small, 1998; Tr?ndle, 2020). The final round of applause can exhibit many interesting dynamics related to their collective enthusiasm for the performance (Lupyan & Rifkin, 2003), the local applause culture, and what is on stage during the clapping. Quantitative empirical study of group clapping behaviours has principally depended on participants clapping on request in laboratory settings or A/V recordings from concerts (Neda, 2000) and presentations (Mann et al., 2013). To study the coordination involved in this collective behaviour, we need accurate measurements of individuals clapping voluntarily in a real concert setting. To describe how the appreciative audience members adjust their clapping to each other and the action on stage during the final round of applause, demonstrating their participation at a concert¡¯s end. * After the Danish String Quartet (DSQ) performed their last piece at the Music Lab Copenhagen Concert, the audience clapped continuously for nearly two minutes. During that time, the musicians stood and bowed, had scientific instruments removed from their bodies, left the stage, returned to bow again, and finally left the stage for good. The clapping action of individual participants in this concert experiment was captured by a mobile phone on their chests, and these recordings show how individuals¡¯ clapping contributed to the collective effect shared with the musicians. Through the final applause interval, 70 devices captured clear clap sequences, representing over half of the audience at this chamber performance. In some ways, their applause followed expected patterns for a concert audience. They began to applaud over a very short time interval (Mann et al., 2013), more than half starting within less than a second of each other. After 20 s of independent clapping at rates from under 120 BMP to over 200 BPM, the participants shifted to clapping together on a shared beat, a practice that is common for Danish audiences. This group maintained synchrony for over a minute while steadily accelerating from around 158 BMP to 176 BMP, an expected consequence of mutual adaptation during group clapping (Thomson et al., 2018). The coordinated action was strongest while the musicians were on the stage but a subset of independent clapping broke out while the audience waited for the performers to return for their final round of bows. Participants¡¯ claps were evaluated from two perspectives: the alignment of claps, reflecting the dominant shift from independence to coordination, and the distribution of participants¡¯ clapping rates over time. Despite some measurement challenges, the shift from independent to coordinate clapping emerges strongly from participants¡¯ movements, with the median rate of clapping slowing until a dominant beat takes hold. Individuals¡¯ clap sequences confirm that the independent clapping at the start of the applause is a result of individual participants clapping isochronously at their own rate, separate in rate and phase from their neighbours in the hall. When the audience claps together, they are voluntarily adjusting to the dominant rate and phase of the people in the hall, with little change in the quality of their isochronous clapping action. Drift in the synchronised clapping rate reflects mutual attentiveness while variation in the number of participants contributing to the coordinated claps suggests differences in applause strategy. Many participants opted to coordinate with their peers while some seemed to prioritise reacting to the musicians.

• Upham, Finn; Park, So Yeon & Lee, Jin Ha (2022). Concert tweeting: Textures of fan activity on twitter during Sowoozoo. Show summary

  Live-streamed BTS concerts have become a familiar fan activity for ARMY. While the broadcasting platform gives fans reliable access to the show, these remote audiences are also using alternative platforms to connect with each other. We look at Twitter activity using the #SOWOOZOO tag around the muster concerts in 2021 to understand how and when ARMY twitter interacted during the performance. Liking and retweeting activity followed the attentional demands of the concert live stream. We observed greater fan engagement during the less active times of Sowoozoo (e.g., when BTS were off stage or talking for long periods). This suggested fans did not want to look away from the exciting musical performances but did read and engage with circulating tweets when the program allowed, like in-person audience members checking in with their friends between sets. However, most original posts to the hashtag had a different pattern, reflecting fans¡¯ needs to shout about what was on stage regardless of whether anyone was looking to read their reactions in real time. Although they contained highly relatable expressions of the tweeter¡¯s own strong feelings of love and excitement, these reaction posts received relatively little engagement, buried quickly in the continued heavy flow on ARMY twitter¡¯s timelines. Instead, the most popular tweets to share and like had high quality information like video clips and translations.Tweets in the middle range of engagement often included both expressions of excitement and valuable media. Disinterested in the limited interaction opportunities within the concert streaming platform, these fans adapted Twitter to serve multiple objectives, from screaming their feelings in a virtual crowd and to collecting and sharing the concert-related content they loved.

• Upham, Finn & Oddekalv, Kjell Andreas (2022). Tapping into the flow: Proprioceptive interaction with rap in Rhythm Hive. Show summary

  Rhythm Hive is a mobile rhythm game released that features music by the Hybe bands BTS, Enhyphen, and TXT. With examples of gameplay, we describe how this app gives fans access to new levels of appreciation for rap verses, particularly for non-korean speakers and non-rappers. Interactive proprioception, a concept by game designer Matt Bloch (Miller, 2017), refers to changes in a player's perception of music as they learn movements to it. Choreography draws attention to coincidental elements of a track, allowing players to intimately and viscerally act with a recorded performance. Players¡¯ actions in Rhythm Hive principally parallel the voiced tracks, the members¡¯ contributions, through dense combinations of finger taps, holds, and slides. Taps to sung phrases often follow melodic pitch but the gestures for rap verses must negotiate different criteria for choosing which syllables to tap to and what shape to give mostly-monotone 16th note runs. In a rap flow - here defined as the rhythm of the words and rhymes in a piece of rap music - the rhythmic structure is created by the placement of the rapped syllables, the stress patterns of said syllables, rhymes and phrase boundaries within a music-rhythmic context. Flow, together with delivery (the way the rhythms are performed) and lyrical content constitute the rap¡¯s expression. In Rhythm Hive, players kinesthetically experience structural features of rap flows: how rhythmic patterns are repeated with slight but significant differences, how long phrases cross music-metrical boundaries to build up tension towards an eventual release, and how phrase-endings with rhymes and punchlines are significant gestures. Comparing Easy and Hard mode sequences to select excerpts of the BTS discography, we show how gameplay illuminates the artistry in these verses by tapping along with and therefore into the flow.

• Upham, Finn (2022). Audience reconstructed: Fan interactions on twitter during livestreamed BTS concerts.
• Swarbrick, Dana; Upham, Finn; McAdams, Stephen; Trainor, Laurel & Merrill, Julia (2022). Concert Experiment Research - Reflections on Past and Future. Show summary

  In the twenty years since the seminal concert experiments conducted by Stephen McAdams, many music cognition labs have invested resources and expertise into studying music in a concert setting. Such projects often involve collaborations between teams of researchers and artists, new technologies, challenging measurement conditions, and creative analysis strategies. As music science events, they can be very attractive to the popular press and yet difficult to report in academic circles. Complications from their bid for greater ecological validity can look like a weakness by laboratory standards, but such challenges to common practices and theories may also be a strength of this distinct research paradigm. In this moderated panel, we gather researchers with a range of experiences in researching musical concerts to share lessons learned and hopeful directions for this experimental paradigm. What should we expect to capture from musicians and audiences in these conditions? What research questions can and should be investigated with live performance and joint spectatorship? Can today¡¯s technologies improve the methods used to conduct these studies? The symposium will include short presentations on past research with a focus on methods and strategy, some discussion of the main questions between the panelists, and a substantial portion of time devoted to discussion with the audience, as experience with concert studies is wider than the published record suggests. Specifically, the panel will consist of three sections: 1) introductions of the panelists¡¯ expertise on concert experiment research, 2) question and answer period between panelists, and 3) an audience question period. The panel will consist of four leading researchers in the field of concert studies and it will be hosted by a doctoral researcher with expertise in the field.

• Swarbrick, Dana; Upham, Finn; Erdem, Cagri; Jensenius, Alexander Refsum & Vuoskoski, Jonna Katariina (2022). Measuring Virtual Audiences with The MusicLab App: Proof of Concept. Show summary

  We present a proof of concept by using the mobile application MusicLab to measure motion during a livestreamed concert and examining its relation to musical features. With the MusicLab App, participants¡¯ own smartphones¡¯ inertial measurement unit (IMU) sensors can be leveraged to record their motion and their subjective experiences collected through survey responses. The MusicLab Lock-down Rave was an Algorave (live-coded dance music) livestreamed concert featuring prolific performers Renick Bell and Khoparzi. They livestreamed for an international audience who wore their smartphones with the MusicLab App while they listened/danced to the performances. From their acceleration, we computed quantity of motion and compared it to musical features that have previously been associated with music-related motion, namely pulse clarity and low and high spectral flux. By encountering challenges and implementing improvements, the MusicLab App has become a useful tool for researching music-related motion.

• Swarbrick, Dana; Upham, Finn; Erdem, Cagri; Burnim, Kayla & Jensenius, Alexander Refsum (2021). The MusicLab App ¨C Exploring the usage of mobile accelerometry to measure audience movement and respiration.
• Swarbrick, Dana; Upham, Finn; Erdem, Cagri; Burnim, Kayla & Jensenius, Alexander Refsum (2021). MusicLab Algorave ¨C An exploratory study examining the usage of mobile accelerometry to measure movements of a virtual concert audience.
• Upham, Finn; Zelechowska, Agata; Gonzalez, Victor & Jensenius, Alexander Refsum (2021). Quiet Breathing to Heard Music.

View all works in Cristin