@@ -54,3 +54,5 @@ autoconf/autom4te.cache

.vs

# clangd index

.clangd

# patch reject

*.rej

@@ -1,18 +1,12 @@

The LLVM Compiler Infrastructure

================================

This directory and its subdirectories contain source code for LLVM,

a toolkit for the construction of highly optimized compilers,

optimizers, and runtime environments.

LLVM is open source software. You may freely distribute it under the terms of

the license agreement found in LICENSE.txt.

Please see the documentation provided in docs/ for further

assistance with LLVM, and in particular docs/GettingStarted.rst for getting

started with LLVM and docs/README.txt for an overview of LLVM's

documentation setup.

If you are writing a package for LLVM, see docs/Packaging.rst for our

suggestions.

This repository contains a fork of LLVM (see https://llvm.org) with patches

towards supporting the RISC-V vector extension (see

https://github.com/riscv/riscv-v-spec/). This is very much a work in progress.

See `docs/RISCVVectorCodegen.md` for some documentation.

These patches are regularly rebased to track LLVM trunk, which means commit

hashes will change. To mitigate the annoyance that causes, development always

happens on a branch named `rvv-<date>` where `<date>` is the date of the last

rebase, and on the next rebase this branch is left alone and a new branch is

created. The downside of this is that you will have to check the list of

branches to find the latest version.

@@ -0,0 +1,69 @@

Code Generation for the RISC-V Vector Extension

===============================================

The vector processing capabilities added by the vector extension are very flexible, but this flexibility introduces some unique challenges in generating correct code for it, let alone optimized code. This document gives a bird's eye view of how vector values and operations are handled throughout the RISCV backend to solve these challenges.

## Context

The LLVM IR we receive as input explicitly operates on vectors of unknown length (e.g., `<scalable 1 x i32>`). The number of elements in an IR vector can be a compile-time integer multiple of the unknown `vscale` (e.g. `<scalable 4 x i32>`), but only a multiple of 1 is legal in the RISCV backend. The unknown factor of the vector element count (`vscale`) corresponds directly to the RVV *Maximum Vector Length* (MVL).

The IR occasionally uses RISCV-specific intrinsics that take and return the *active vector length* as an integer value, as in this fragment:

```

%vl = call i32 @llvm.riscv.setvl(i32 %n)

%a = call <scalable 1 x i32> @llvm.riscv.vlw(i32* %p, i32 %vl)

```

However, there are also target-independent operations such as `add <scalable 1 x i32>` that have no such concept.

## Instruction Selection

Instruction selection ignores the vector unit configuration entirely: it operates as if we had a full hard-wired set of regular vector registers with some unknown-but-fixed number of elements. This is done by using pseudo-instructions with suffix `_ImpConf` (*imp*licit *conf*iguration).

In general these instructions would omit an important dependency and therefore open us up to miscompilations, but during instruction selection it's fine because we select no instructions that *change* the configuration (that comes later).

Another concern is the active vector length, which is an operand to most vector instructions we want to select. For IR instructions such as `add <scalable 1 x i32>` that have no VL operand in IR, we use MVL for that operation (and hope to optimize it later). For RISCV-specific intrinsics that handle the active vector length, we just use the integer passed to that intrinsic.

## Vector length register

We represent the VL CSR as an ordinary physical register that can hold an XLEN-sized integer (`XLenVT`). There is a register class, `VLR`, that contains only VL. Most vector instructions have `VLR` inputs, `setvl` has a `VLR` output (along with the GPR result).

This register class is naturally not the preferred one for selecting integer code, but copies between it and GPRs are possible via CSR reads and writes.

> Note: Still need to decide on a solution for VLR virtregs with overlapping live ranges. As-is they cause crashes during register allocation.

## Vector unit configuration CSR

Ẃe represent the vector unit configuration CSR as an unallocatable, permanently reserved physical register.

It is implicitly used by most vector instructions, and instructions that change the configuration define this register implicitly.

There are no corresponding virtual registers, it's a physical register operand from the very start.

## Selecting the active vector length

> Note: This part is not implemented yet.

RISC-V vector instructions are generally controlled by the VL register and only process the lanes up to VL and zero the remaining elements of the destination. This does not match LLVM IR semantics, where many vector operations operate on the full vector and prediction is often applied as a separate operation afterwards.

For correctness, the RISCV backend must therefore be prepared to generate code that sets VL to MAXVL, i.e., operates on full vectors. This is particularly relevant for register copies and spill/fill code, but may also sometimes be required for arithmetic operations (though it shouldn't be required for IR produced by a V-aware vectorizer).

Nevertheless, we really want to use VL in the natural way. Thus we perform a "demanded elements" analysis which symbolically computes .

This is only correct for side-effect-free operations, but operations with side effects need to be "inherently predicated" in the IR already (e.g., consider `@llvm.masked.load` versus a regular vector load).

## Deciding the configuration

The configuration is decided before register allocation, because register allocation needs complete knowledge of the physical register field.

Integrating these decisions into register allocation would be even better, but it's not clear how to do that (and it would be likely a huge project in any case).

By looking at the live intervals, proportions of data type widths among them, etc. we can at least try to heuristically find a good fit.

In the future we should try to separate the function body into several regions between which we can safely change the configuration (at minimum this means no vector values live between them, not even in memory).

## Scalar operations in vector registers

Details TBD, see https://lists.llvm.org/pipermail/llvm-dev/2018-October/126733.html for some thoughts on this

## Emitting configuration instructions

Currently the vector unit is configured in the prologue and disabled in the epilogue.

In the future, especially once we start to support more than one configurations per function, we may insert configuration changes earlier, and also place them closer to the code that actually uses the vector unit.

@@ -2032,6 +2032,8 @@ class ShuffleVectorInst : public Instruction {

static void getShuffleMask(const Constant *Mask,

SmallVectorImpl<int> &Result);

static bool getShuffleMask(Value *Mask, SmallVectorImpl<int> &Result);

/// Return the mask for this instruction as a vector of integers. Undefined

/// elements of the mask are returned as -1.

void getShuffleMask(SmallVectorImpl<int> &Result) const {

@@ -285,6 +285,8 @@ def llvm_v2f64_ty : LLVMType<v2f64>; // 2 x double

def llvm_v4f64_ty : LLVMType<v4f64>; // 4 x double

def llvm_v8f64_ty : LLVMType<v8f64>; // 8 x double

def llvm_nxv1i32_ty : LLVMType<nxv1i32>; // scalable 1 x i32

def llvm_vararg_ty : LLVMType<isVoid>; // this means vararg here

//===----------------------------------------------------------------------===//

@@ -1230,6 +1232,8 @@ let IntrProperties = [IntrNoMem, IntrWillReturn] in {

[llvm_anyvector_ty]>;

def int_experimental_vector_reduce_fmin : Intrinsic<[LLVMVectorElementType<0>],

[llvm_anyvector_ty]>;

def int_experimental_vector_splatvector : Intrinsic<[LLVMVectorElementType<0>],

[llvm_anyvector_ty]>;

}

//===---------- Intrinsics to control hardware supported loops ----------===//

@@ -65,4 +65,51 @@ def int_riscv_masked_cmpxchg_i64

llvm_i64_ty, llvm_i64_ty],

[IntrArgMemOnly, NoCapture<0>, ImmArg<4>]>;

//===----------------------------------------------------------------------===//

// Vector extension

def int_riscv_setvl : Intrinsic<[llvm_i32_ty], [llvm_i32_ty], [IntrNoMem]>;

def int_riscv_vadd : Intrinsic<[llvm_nxv1i32_ty],

[llvm_nxv1i32_ty, llvm_nxv1i32_ty, llvm_i32_ty],

[IntrNoMem]>;

def int_riscv_vsub : Intrinsic<[llvm_nxv1i32_ty],

[llvm_nxv1i32_ty, llvm_nxv1i32_ty, llvm_i32_ty],

[IntrNoMem]>;

def int_riscv_vmul : Intrinsic<[llvm_nxv1i32_ty],

[llvm_nxv1i32_ty, llvm_nxv1i32_ty, llvm_i32_ty],

[IntrNoMem]>;

def int_riscv_vand : Intrinsic<[llvm_nxv1i32_ty],

[llvm_nxv1i32_ty, llvm_nxv1i32_ty, llvm_i32_ty],

[IntrNoMem]>;

def int_riscv_vor : Intrinsic<[llvm_nxv1i32_ty],

[llvm_nxv1i32_ty, llvm_nxv1i32_ty, llvm_i32_ty],

[IntrNoMem]>;

def int_riscv_vxor : Intrinsic<[llvm_nxv1i32_ty],

[llvm_nxv1i32_ty, llvm_nxv1i32_ty, llvm_i32_ty],

[IntrNoMem]>;

def int_riscv_vlw : Intrinsic<[llvm_nxv1i32_ty],

[llvm_ptr32_ty, llvm_i32_ty],

[IntrReadMem]>;

def int_riscv_vsw : Intrinsic<[],

[llvm_ptr32_ty, llvm_nxv1i32_ty, llvm_i32_ty],

[IntrWriteMem]>;

def int_riscv_vmpopcnt : Intrinsic<[llvm_i32_ty],

[llvm_nxv1i32_ty, llvm_i32_ty],

[IntrNoMem]>;

def int_riscv_vmfirst : Intrinsic<[llvm_i32_ty],

[llvm_nxv1i32_ty, llvm_i32_ty],

[IntrNoMem]>;

} // TargetPrefix = "riscv"

@@ -1545,6 +1545,17 @@ SDValue SelectionDAGBuilder::getValueImpl(const Value *V) {

Op = DAG.getConstantFP(0, getCurSDLoc(), EltVT);

else

Op = DAG.getConstant(0, getCurSDLoc(), EltVT);

if (VT.isScalableVector()) {

auto INum = DAG.getConstant(Intrinsic::experimental_vector_splatvector,

getCurSDLoc(), MVT::i32);

auto Splat = DAG.getNode(ISD::INTRINSIC_WO_CHAIN, getCurSDLoc(), VT,

INum, Op);

return Splat;

}

Ops.assign(NumElements, Op);

}

@@ -3538,17 +3549,51 @@ void SelectionDAGBuilder::visitExtractElement(const User &I) {

void SelectionDAGBuilder::visitShuffleVector(const User &I) {

SDValue Src1 = getValue(I.getOperand(0));

SDValue Src2 = getValue(I.getOperand(1));

Value *MaskV = I.getOperand(2);

SDLoc DL = getCurSDLoc();

SmallVector<int, 8> Mask;

ShuffleVectorInst::getShuffleMask(cast<Constant>(I.getOperand(2)), Mask);

unsigned MaskNumElts = Mask.size();

const TargetLowering &TLI = DAG.getTargetLoweringInfo();

EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());

bool IsScalable = VT.isScalableVector();

EVT SrcVT = Src1.getValueType();

unsigned SrcNumElts = SrcVT.getVectorNumElements();

SmallVector<int, 8> Mask;

if (!ShuffleVectorInst::getShuffleMask(MaskV, Mask)) {

SDValue Mask = getValue(I.getOperand(2));

unsigned NumElts = VT.getVectorNumElements();

// We don't currently support variable shuffles on fixed-length vectors

assert(IsScalable && "Non-constant shuffle mask on fixed-length vector");

// We haven't introduced a vector_shuffle_var intrinsic to support shuffles

// where we need to extract or merge vectors.

if (NumElts != SrcNumElts)

llvm_unreachable("Haven't implemented VECTOR_SHUFFLE_VAR intrinsic yet");

// Currently only handling splats of a single value for scalable vectors

if (auto *CMask = dyn_cast<Constant>(MaskV))

if (CMask->isNullValue()) {

// Splat of first element.

auto FirstElt = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL,

SrcVT.getScalarType(), Src1,

DAG.getConstant(0, DL,

TLI.getVectorIdxTy(DAG.getDataLayout())));

auto INum = DAG.getConstant(Intrinsic::experimental_vector_splatvector,

DL, MVT::i32);

auto Splat = DAG.getNode(ISD::INTRINSIC_WO_CHAIN, DL, VT,

INum, FirstElt);

setValue(&I, Splat);

return;

}

llvm_unreachable("Haven't implemented VECTOR_SHUFFLE_VAR intrinsic yet");

return;

}

unsigned MaskNumElts = Mask.size();

if (SrcNumElts == MaskNumElts) {

setValue(&I, DAG.getVectorShuffle(VT, DL, Src1, Src2, Mask));

return;

@@ -797,8 +797,9 @@ Constant *llvm::ConstantFoldExtractElementInstruction(Constant *Val,

if (ConstantInt *CIdx = dyn_cast<ConstantInt>(Idx)) {

// ee({w,x,y,z}, wrong_value) -> undef

if (CIdx->uge(Val->getType()->getVectorNumElements()))

return UndefValue::get(Val->getType()->getVectorElementType());

if (!Val->getType()->getVectorIsScalable())

if (CIdx->uge(Val->getType()->getVectorNumElements()))

return UndefValue::get(Val->getType()->getVectorElementType());

return Val->getAggregateElement(CIdx->getZExtValue());

}

return nullptr;

@@ -810,6 +811,10 @@ Constant *llvm::ConstantFoldInsertElementInstruction(Constant *Val,

if (isa<UndefValue>(Idx))

return UndefValue::get(Val->getType());

// Everything after this point assumes you can iterate across Val.

if (Val->getType()->getVectorIsScalable())

return nullptr;

ConstantInt *CIdx = dyn_cast<ConstantInt>(Idx);

if (!CIdx) return nullptr;

@@ -837,8 +842,9 @@ Constant *llvm::ConstantFoldInsertElementInstruction(Constant *Val,

Constant *llvm::ConstantFoldShuffleVectorInstruction(Constant *V1,

Constant *V2,

Constant *Mask) {

unsigned MaskNumElts = Mask->getType()->getVectorNumElements();

Type *EltTy = V1->getType()->getVectorElementType();

auto *MaskTy = cast<VectorType>(Mask->getType());

auto MaskNumElts = MaskTy->getElementCount();

Type *EltTy = V1->getType()->getVectorElementType();

// Undefined shuffle mask -> undefined value.

if (isa<UndefValue>(Mask))

@@ -847,11 +853,23 @@ Constant *llvm::ConstantFoldShuffleVectorInstruction(Constant *V1,

// Don't break the bitcode reader hack.

if (isa<ConstantExpr>(Mask)) return nullptr;

if (MaskTy->isScalable()) {

// Is splat?

if (Mask->isNullValue()) {

Constant *Zero = Constant::getNullValue(MaskTy->getElementType());

Constant *SplatVal = ConstantFoldExtractElementInstruction(V1, Zero);

// Is splat of zero?

if (SplatVal && SplatVal->isNullValue())

return Constant::getNullValue(VectorType::get(EltTy, MaskNumElts));

}

return nullptr;

}

unsigned SrcNumElts = V1->getType()->getVectorNumElements();

// Loop over the shuffle mask, evaluating each element.

SmallVector<Constant*, 32> Result;

for (unsigned i = 0; i != MaskNumElts; ++i) {

for (unsigned i = 0; i != MaskNumElts.Min; ++i) {

int Elt = ShuffleVectorInst::getMaskValue(Mask, i);

if (Elt == -1) {

Result.push_back(UndefValue::get(EltTy));

@@ -2166,8 +2166,9 @@ Constant *ConstantExpr::getShuffleVector(Constant *V1, Constant *V2,

return FC; // Fold a few common cases.

unsigned NElts = Mask->getType()->getVectorNumElements();

bool Scalable = Mask->getType()->getVectorIsScalable();

Type *EltTy = V1->getType()->getVectorElementType();

Type *ShufTy = VectorType::get(EltTy, NElts);

Type *ShufTy = VectorType::get(EltTy, NElts, Scalable);

if (OnlyIfReducedTy == ShufTy)

return nullptr;

@@ -149,7 +149,7 @@ class ShuffleVectorConstantExpr : public ConstantExpr {

ShuffleVectorConstantExpr(Constant *C1, Constant *C2, Constant *C3)

: ConstantExpr(VectorType::get(

cast<VectorType>(C1->getType())->getElementType(),

cast<VectorType>(C3->getType())->getNumElements()),

cast<VectorType>(C3->getType())->getElementCount()),

Instruction::ShuffleVector,

&Op<0>(), 3) {

Op<0>() = C1;